Compare commits

Author SHA1 Message Date
codex 1a341f1eaa feat: attach calendar metadata to active meetings
PR and Push Build/Test / build-and-test (push) Successful in 15m2s
2026-08-05 11:58:25 +02:00
codex aa42e8edda refactor: tighten tray menu section handling
PR and Push Build/Test / build-and-test (push) Successful in 10m51s
2026-08-04 10:11:43 +02:00
codex 2f12a96688 feat: prioritize finish meeting tray action
PR and Push Build/Test / build-and-test (push) Successful in 11m8s
2026-08-04 09:55:18 +02:00
codex 5d0ae84426 fix: support streaming screenshot OCR 2026-08-03 16:14:01 +02:00
codex b9547ae4c4 fix: recover from microphone disconnects
PR and Push Build/Test / build-and-test (push) Successful in 12m29s
2026-08-03 15:57:30 +02:00
codex 75250f6041 Merge pull request 'Update dependency SQLitePCLRaw.bundle_e_sqlite3 to 3.0.5' (#35) from renovate/sqlitepclraw.bundle_e_sqlite3-3.x into main
PR and Push Build/Test / build-and-test (push) Successful in 10m19s
2026-07-30 15:42:00 +02:00
renovate-bot cd2446f620 Update dependency SQLitePCLRaw.bundle_e_sqlite3 to 3.0.5
PR and Push Build/Test / build-and-test (pull_request) Successful in 10m58s
PR and Push Build/Test / build-and-test (push) Successful in 13m32s
2026-07-28 02:38:03 +00:00
codex e192ae7cd8 fix: support LiteLLM Responses streaming
PR and Push Build/Test / build-and-test (push) Successful in 14m17s
2026-07-27 14:44:32 +02:00
codex 3cadf08fa2 feat: use assistant context as meeting memory
PR and Push Build/Test / build-and-test (push) Successful in 9m34s
2026-07-17 11:26:49 +02:00
Manuel 407db80413 Merge pull request 'Update dependency Microsoft.NET.Test.Sdk to 18.8.1' (#33) from renovate/microsoft.net.test.sdk-18.x into main
PR and Push Build/Test / build-and-test (push) Successful in 13m56s
2026-07-16 18:50:55 +02:00
renovate-bot f0aac40dfb Update dependency Microsoft.NET.Test.Sdk to 18.8.1
PR and Push Build/Test / build-and-test (push) Successful in 9m38s
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m28s
2026-07-16 02:36:52 +00:00
codex 0e0feedad2 Fix YAML escaping for meeting note frontmatter
PR and Push Build/Test / build-and-test (push) Successful in 10m21s
2026-07-15 14:55:37 +02:00
codex 89d81fa4c6 fix: encode spaces in summary image links
PR and Push Build/Test / build-and-test (push) Successful in 11m2s
2026-07-15 14:55:25 +02:00
Manuel dc49bc3330 Merge pull request 'Update dependency System.Drawing.Common to 10.0.10' (#32) from renovate/system.drawing.common-10.x into main
PR and Push Build/Test / build-and-test (push) Successful in 9m38s
2026-07-15 13:29:06 +02:00
Manuel aff3528406 Merge pull request 'Update dependency Microsoft.EntityFrameworkCore.Sqlite to 10.0.10' (#31) from renovate/microsoft.entityframeworkcore.sqlite-10.x into main
PR and Push Build/Test / build-and-test (push) Failing after 9m50s
2026-07-15 13:27:51 +02:00
Manuel d92da18d08 Merge pull request 'Update dependency Microsoft.AspNetCore.Mvc.Testing to 10.0.10' (#30) from renovate/microsoft.aspnetcore.mvc.testing-10.x into main
PR and Push Build/Test / build-and-test (push) Successful in 9m41s
2026-07-15 13:27:20 +02:00
renovate-bot ec59464340 Update dependency System.Drawing.Common to 10.0.10
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m19s
PR and Push Build/Test / build-and-test (push) Successful in 9m33s
2026-07-15 09:58:37 +00:00
renovate-bot 5667102272 Update dependency Microsoft.EntityFrameworkCore.Sqlite to 10.0.10
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m33s
PR and Push Build/Test / build-and-test (push) Successful in 9m9s
2026-07-15 02:35:11 +00:00
renovate-bot 2f62ed467a Update dependency Microsoft.AspNetCore.Mvc.Testing to 10.0.10
PR and Push Build/Test / build-and-test (push) Successful in 9m8s
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m15s
2026-07-15 02:35:02 +00:00
Manuel 0439a819b2 Merge pull request 'Update actions/checkout action to v7' (#22) from renovate/actions-checkout-7.x into main
PR and Push Build/Test / build-and-test (push) Successful in 11m32s
Reviewed-on: #22
2026-07-05 01:43:27 +02:00
Manuel 2cfe4e4ef0 Merge pull request 'Update dependency Microsoft.Agents.AI.OpenAI to 1.13.0' (#28) from renovate/microsoft.agents.ai.openai-1.x into main
PR and Push Build/Test / build-and-test (push) Successful in 11m11s
Reviewed-on: #28
2026-07-05 01:39:59 +02:00
renovate-bot b5ccf2125c Update dependency Microsoft.Agents.AI.OpenAI to 1.13.0
PR and Push Build/Test / build-and-test (push) Successful in 10m18s
PR and Push Build/Test / build-and-test (pull_request) Successful in 10m8s
2026-07-04 02:33:36 +00:00
codex 0a7fc240d9 Merge pull request 'Update dependency NCalcSync to 6.4.0' (#29) from renovate/ncalcsync-6.x into main
PR and Push Build/Test / build-and-test (push) Successful in 9m38s
2026-07-03 13:26:22 +02:00
renovate-bot d81c2731da Update actions/checkout action to v7
PR and Push Build/Test / build-and-test (push) Failing after 9m39s
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m50s
2026-07-03 02:33:23 +00:00
renovate-bot aa8ac26bbb Update dependency NCalcSync to 6.4.0
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m46s
PR and Push Build/Test / build-and-test (push) Successful in 9m42s
2026-07-03 02:33:17 +00:00
codex 6ee2d0ea08 Merge pull request 'Update dependency NCalcSync to 6.3.3' (#27) from renovate/ncalcsync-6.x into main
PR and Push Build/Test / build-and-test (push) Successful in 14m17s
2026-07-02 10:10:32 +02:00
codex e8742f544d Merge pull request 'Update dependency YamlDotNet to 18.1.0' (#26) from renovate/yamldotnet-18.x into main
PR and Push Build/Test / build-and-test (push) Successful in 10m12s
2026-07-02 10:10:07 +02:00
renovate-bot 18bed0532a Update dependency NCalcSync to 6.3.3
PR and Push Build/Test / build-and-test (push) Successful in 9m54s
PR and Push Build/Test / build-and-test (pull_request) Successful in 10m1s
2026-07-02 02:32:16 +00:00
codex 8e2f266e66 Close stale transcript workflow follow-up
PR and Push Build/Test / build-and-test (push) Successful in 9m35s
2026-07-01 11:44:07 +02:00
codex 5c67738939 Add attendee transformation workflows 2026-07-01 11:10:36 +02:00
codex 92e359646b Update meeting summary agent UI 2026-07-01 10:30:28 +02:00
renovate-bot c6b2add4dd Update dependency YamlDotNet to 18.1.0
PR and Push Build/Test / build-and-test (push) Successful in 9m19s
PR and Push Build/Test / build-and-test (pull_request) Successful in 9m21s
2026-06-27 02:30:48 +00:00
96 changed files with 4130 additions and 837 deletions
+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@v7
- name: Setup .NET
uses: actions/setup-dotnet@v5
@@ -102,6 +102,23 @@ public sealed class AudioMixingTests
Assert.Equal(2_000, BitConverter.ToInt16(chunks[0].Pcm));
}
[Fact]
public async Task CompositeAudioSourceKeepsSystemAudioWhileMicrophoneIsRecovering()
{
var microphone = new WaitingAudioSource();
var system = new FixedAudioSource(Pcm16(10_000));
var source = CreateSource(microphone, system);
using var cancellation = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await using var chunks = source
.CaptureAsync(new MeetingAssistantOptions(), cancellation.Token)
.GetAsyncEnumerator(cancellation.Token);
Assert.True(await chunks.MoveNextAsync());
Assert.Equal(10_000, BitConverter.ToInt16(chunks.Current.Pcm));
await cancellation.CancelAsync();
}
[Fact]
public void AdaptiveEchoCancellerReducesEchoFromMicrophoneSignal()
{
@@ -221,6 +238,16 @@ public sealed class AudioMixingTests
}
}
private sealed class WaitingAudioSource : IMeetingAudioSource
{
public async IAsyncEnumerable<AudioChunk> CaptureAsync(
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
yield break;
}
}
private sealed class OptionsCapturingAudioSource : IMeetingAudioSource
{
private readonly AudioChunk chunk;
@@ -123,6 +123,49 @@ public sealed class CalendarRecordingPromptSchedulerTests
Assert.Equal(["stop", "start"], harness.Recorder.Commands);
}
[Fact]
public async Task ActiveRecordingPromptAttachesTheSelectedAppointmentMetadataWithoutRestarting()
{
var harness = CreateHarness(isRecording: true, autoAcceptPrompts: false);
var firstMetadata = new MeetingMetadata(
"First planning",
["Ada"],
"First agenda",
DateTimeOffset.Parse("2026-06-03T10:30:00+00:00"));
var secondMetadata = new MeetingMetadata(
"Second planning",
["Grace"],
"Second agenda",
DateTimeOffset.Parse("2026-06-03T10:30:00+00:00"));
var firstMeeting = CreateMeeting(
harness.Clock,
id: "teams-first",
subject: "First planning",
metadata: firstMetadata);
var secondMeeting = CreateMeeting(
harness.Clock,
id: "teams-second",
subject: "Second planning",
metadata: secondMetadata);
harness.Provider.Meetings = [firstMeeting, secondMeeting];
await harness.Scheduler.SyncOnceAsync(CancellationToken.None);
harness.Clock.Now = firstMeeting.Start;
await harness.Scheduler.CheckDuePromptsAsync(CancellationToken.None);
await harness.PromptService.RespondAsync(
secondMeeting,
MeetingStartPromptResponse.AttachMetadataToCurrentMeeting);
Assert.All(
harness.PromptService.PromptRequests,
request => Assert.True(request.CanAttachToCurrentMeeting));
Assert.Equal(["attach-metadata"], harness.Recorder.Commands);
Assert.Same(secondMetadata, harness.Recorder.AttachedMetadata.Single());
Assert.Equal(0, harness.Recorder.StopCount);
Assert.Equal(0, harness.Recorder.StartCount);
Assert.True(harness.Recorder.CurrentStatus.IsRecording);
}
[Fact]
public async Task CanceledCachedMeetingDoesNotPromptRecording()
{
@@ -302,14 +345,17 @@ public sealed class CalendarRecordingPromptSchedulerTests
this.autoAccept = autoAccept;
}
public List<CalendarMeeting> PromptedMeetings { get; } = [];
public List<MeetingStartPromptRequest> PromptRequests { get; } = [];
public IReadOnlyList<CalendarMeeting> PromptedMeetings =>
PromptRequests.Select(request => request.Meeting).ToList();
public async Task ShowPromptAsync(
MeetingStartPromptRequest request,
Func<MeetingStartPromptResponse, CancellationToken, Task> handleResponseAsync,
CancellationToken cancellationToken)
{
PromptedMeetings.Add(request.Meeting);
PromptRequests.Add(request);
pendingPrompts.Add(new PendingPrompt(request.Meeting, handleResponseAsync));
if (autoAccept)
{
@@ -349,6 +395,8 @@ public sealed class CalendarRecordingPromptSchedulerTests
public List<MeetingMetadata?> StartMetadata { get; } = [];
public List<MeetingMetadata> AttachedMetadata { get; } = [];
public Task<RecordingStatus> StartAsync(CancellationToken cancellationToken)
{
return StartRecordingAsync(null);
@@ -380,6 +428,15 @@ public sealed class CalendarRecordingPromptSchedulerTests
return Task.FromResult(CurrentStatus);
}
public Task<RecordingStatus> AttachMetadataToCurrentMeetingAsync(
MeetingMetadata metadata,
CancellationToken cancellationToken)
{
Commands.Add("attach-metadata");
AttachedMetadata.Add(metadata);
return Task.FromResult(CurrentStatus);
}
private static RecordingStatus Status(bool isRecording)
{
return new RecordingStatus(
@@ -1,103 +1,253 @@
using MeetingAssistant.Summary;
using Microsoft.Extensions.AI;
using System.Net;
using System.Text;
using System.Text.Json;
namespace MeetingAssistant.Tests;
public sealed class LiteLlmResponsesChatClientTests
{
[Fact]
public void ParserIgnoresReasoningItemsWithNullStatusAndReadsText()
public async Task ClientAssemblesStreamedTextResponseWithMetadataAndUsage()
{
const string json = """
var handler = new SequencedHttpMessageHandler(
new HttpResponseMessage(HttpStatusCode.OK)
{
"id": "resp_test",
"created_at": 1779147100,
"model": "gpt-5.5-2026-04-23",
"output": [
{
"type": "reasoning",
"summary": [],
"status": null
},
{
"type": "message",
"role": "assistant",
"status": "completed",
"content": [
{
"type": "output_text",
"text": "OK"
}
]
}
]
}
""";
Content = new StringContent(
"""
data: {"type":"response.created","response":{"id":"resp_stream","created_at":1779147100,"model":"gpt-5.5","object":"response","output":[],"parallel_tool_calls":true,"status":"in_progress","store":false},"sequence_number":0}
var response = LiteLlmResponsesChatClient.ParseResponseJson(json);
data: {"type":"response.output_item.added","output_index":0,"item":{"id":"msg_stream","type":"message","status":"in_progress","content":[],"role":"assistant"},"sequence_number":1}
Assert.Equal("OK", response.Text);
Assert.Equal("resp_test", response.ResponseId);
Assert.Equal("gpt-5.5-2026-04-23", response.ModelId);
data: {"type":"response.content_part.added","item_id":"msg_stream","output_index":0,"content_index":0,"part":{"type":"output_text","annotations":[],"text":""},"sequence_number":2}
data: {"type":"response.output_text.delta","item_id":"msg_stream","output_index":0,"content_index":0,"delta":"Streamed OK","sequence_number":3}
data: {"type":"response.output_text.done","item_id":"msg_stream","output_index":0,"content_index":0,"text":"Streamed OK","sequence_number":4}
data: {"type":"response.content_part.done","item_id":"msg_stream","output_index":0,"content_index":0,"part":{"type":"output_text","annotations":[],"text":"Streamed OK"},"sequence_number":5}
data: {"type":"response.output_item.done","output_index":0,"item":{"id":"msg_stream","type":"message","status":"completed","content":[{"type":"output_text","annotations":[],"text":"Streamed OK"}],"role":"assistant"},"sequence_number":6}
data: {"type":"response.completed","response":{"id":"resp_stream","created_at":1779147100,"model":"gpt-5.5","object":"response","output":[],"parallel_tool_calls":true,"status":"completed","store":false,"usage":{"input_tokens":12,"output_tokens":3,"total_tokens":15}},"sequence_number":7}
data: [DONE]
""",
Encoding.UTF8,
"text/event-stream")
});
using var client = CreateClient(handler, reconnectionAttempts: 0);
var response = await client.GetResponseAsync([new ChatMessage(ChatRole.User, "reply")]);
Assert.Equal("Streamed OK", response.Text);
Assert.Equal("resp_stream", response.ResponseId);
Assert.Equal("gpt-5.5", response.ModelId);
Assert.Equal(DateTimeOffset.FromUnixTimeSeconds(1779147100), response.CreatedAt);
Assert.NotNull(response.Usage);
Assert.Equal(12, response.Usage.InputTokenCount);
Assert.Equal(3, response.Usage.OutputTokenCount);
Assert.Equal(15, response.Usage.TotalTokenCount);
}
[Fact]
public void ParserReadsFunctionCalls()
public async Task ClientAssemblesStreamedFunctionCall()
{
const string json = """
var handler = new SequencedHttpMessageHandler(
new HttpResponseMessage(HttpStatusCode.OK)
{
"output": [
{
"type": "function_call",
"call_id": "call_1",
"name": "write_summary",
"arguments": "{\"markdown\":\"# Summary\\nDone\"}",
"status": "completed"
}
]
}
""";
Content = new StringContent(
"""
data: {"type":"response.output_item.done","output_index":0,"item":{"type":"function_call","call_id":"call_stream","name":"write_summary","arguments":"{\"markdown\":\"# Summary\\nDone\"}","status":"completed"}}
var response = LiteLlmResponsesChatClient.ParseResponseJson(json);
data: {"type":"response.completed","response":{"id":"resp_stream","model":"gpt-5.5","output":[]}}
data: [DONE]
""",
Encoding.UTF8,
"text/event-stream")
});
using var client = CreateClient(handler, reconnectionAttempts: 0);
var response = await client.GetResponseAsync([new ChatMessage(ChatRole.User, "write summary")]);
var call = Assert.IsType<FunctionCallContent>(Assert.Single(response.Messages[0].Contents));
Assert.Equal("call_1", call.CallId);
Assert.Equal("call_stream", call.CallId);
Assert.Equal("write_summary", call.Name);
Assert.NotNull(call.Arguments);
Assert.Equal("# Summary\nDone", call.Arguments["markdown"]);
Assert.Equal("# Summary\nDone", call.Arguments["markdown"]?.ToString());
}
[Fact]
public void ParserReadsUsage()
public async Task ClientUsesNonStreamingResponsesWhenConfigured()
{
const string json = """
{
"usage": {
"input_tokens": 123,
"output_tokens": 45,
"total_tokens": 168
},
"output": [
var handler = new RecordingHttpMessageHandler(_ => new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(
"""
{
"type": "message",
"content": [
"id": "resp_nonstream",
"created_at": 1779147100,
"model": "gpt-5.5",
"object": "response",
"output": [
{
"type": "output_text",
"text": "OK"
"id": "msg_nonstream",
"type": "message",
"status": "completed",
"content": [
{
"type": "output_text",
"annotations": [],
"text": "Non-streamed OK"
}
],
"role": "assistant"
}
]
],
"parallel_tool_calls": true,
"status": "completed",
"store": false,
"usage": {
"input_tokens": 12,
"output_tokens": 4,
"total_tokens": 16
}
}
]
}
""";
""",
Encoding.UTF8,
"application/json")
});
using var client = CreateClient(handler, reconnectionAttempts: 0, useStreaming: false);
var response = LiteLlmResponsesChatClient.ParseResponseJson(json);
var response = await client.GetResponseAsync([new ChatMessage(ChatRole.User, "reply")]);
Assert.NotNull(response.Usage);
Assert.Equal(123, response.Usage.InputTokenCount);
Assert.Equal(45, response.Usage.OutputTokenCount);
Assert.Equal(168, response.Usage.TotalTokenCount);
Assert.Equal("Non-streamed OK", response.Text);
Assert.Contains("\"stream\":false", Assert.Single(handler.RequestBodies));
}
[Fact]
public async Task ClientSendsChatMessagesAsResponsesMessageItems()
{
var handler = new RecordingHttpMessageHandler(_ => CreateStreamedTextResponse("Done."));
using var client = CreateClient(handler, reconnectionAttempts: 0);
await client.GetResponseAsync(
[
new ChatMessage(ChatRole.User, "write summary"),
new ChatMessage(ChatRole.Assistant, "I will inspect the meeting.")
]);
using var request = JsonDocument.Parse(Assert.Single(handler.RequestBodies));
var inputItems = request.RootElement.GetProperty("input").EnumerateArray().ToArray();
Assert.Equal(2, inputItems.Length);
Assert.All(inputItems, item => Assert.Equal("message", item.GetProperty("type").GetString()));
Assert.Equal("user", inputItems[0].GetProperty("role").GetString());
Assert.Equal(
"input_text",
Assert.Single(inputItems[0].GetProperty("content").EnumerateArray())
.GetProperty("type")
.GetString());
Assert.Equal("assistant", inputItems[1].GetProperty("role").GetString());
Assert.Equal(
"output_text",
Assert.Single(inputItems[1].GetProperty("content").EnumerateArray())
.GetProperty("type")
.GetString());
Assert.DoesNotContain("\"type\":\"unknown\"", handler.RequestBodies[0]);
}
[Fact]
public async Task AgentLoopReturnsMalformedFunctionArgumentsWithoutInvokingTool()
{
var responses = new Queue<HttpResponseMessage>(
[
new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(
"""
data: {"type":"response.output_item.done","output_index":0,"item":{"type":"function_call","call_id":"call_invalid","name":"write_summary","arguments":"{not-json","status":"completed"}}
data: {"type":"response.completed","response":{"id":"resp_invalid","model":"gpt-5.5","output":[]}}
data: [DONE]
""",
Encoding.UTF8,
"text/event-stream")
},
CreateStreamedTextResponse("Recovered.")
]);
var handler = new RecordingHttpMessageHandler(_ => responses.Dequeue());
var toolInvoked = false;
var tool = AIFunctionFactory.Create(
(string markdown) =>
{
toolInvoked = true;
return markdown;
},
"write_summary",
"Writes a summary.");
using var innerClient = CreateClient(handler, reconnectionAttempts: 0);
using var functionClient = innerClient
.AsBuilder()
.UseFunctionInvocation(
loggerFactory: null,
client => client.FunctionInvoker = FunctionInvocationGuard.InvokeAsync)
.Build();
var response = await functionClient.GetResponseAsync(
[new ChatMessage(ChatRole.User, "write summary")],
new ChatOptions { Tools = [tool] });
Assert.False(toolInvoked);
Assert.Equal("Recovered.", response.Text);
Assert.Equal(2, handler.RequestBodies.Count);
Assert.Contains("invalid_tool_arguments", handler.RequestBodies[1]);
Assert.Contains("call_invalid", handler.RequestBodies[1]);
}
[Fact]
public async Task ClientReportsVisibleReasoningSummariesSeparatelyFromResponseText()
{
var handler = new SequencedHttpMessageHandler(
CreateStreamedTextResponse(
"Done.",
"Checked the configured rules file.",
"Prepared a targeted update."));
var reasoningSummaries = new List<string>();
using var client = CreateClient(
handler,
reconnectionAttempts: 0,
reasoningSummaryChanged: reasoningSummaries.Add);
var response = await client.GetResponseAsync([new ChatMessage(ChatRole.User, "update rules")]);
Assert.Equal("Done.", response.Text);
Assert.Equal(
[
"Checked the configured rules file.",
"Prepared a targeted update."
],
reasoningSummaries);
}
[Fact]
public async Task ClientDoesNotFailResponseWhenReasoningSummaryCallbackFails()
{
var handler = new SequencedHttpMessageHandler(
CreateStreamedTextResponse("Done.", "Checked the configured rules file."));
using var client = CreateClient(
handler,
reconnectionAttempts: 0,
reasoningSummaryChanged: _ => throw new InvalidOperationException("UI failed"));
var response = await client.GetResponseAsync([new ChatMessage(ChatRole.User, "update rules")]);
Assert.Equal("Done.", response.Text);
}
[Fact]
@@ -108,24 +258,7 @@ public sealed class LiteLlmResponsesChatClientTests
{
Content = new StringContent("Internal Server Error")
},
new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("""
{
"output": [
{
"type": "message",
"content": [
{
"type": "output_text",
"text": "Done."
}
]
}
]
}
""")
});
CreateStreamedTextResponse("Done."));
var retryCount = 0;
using var client = CreateClient(handler, reconnectionAttempts: 1, retrying: () => retryCount++);
@@ -156,24 +289,7 @@ public sealed class LiteLlmResponsesChatClientTests
[Fact]
public async Task ClientSendsUserInitiatorOnceThenAgentInitiator()
{
var handler = new RecordingHttpMessageHandler(_ => new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("""
{
"output": [
{
"type": "message",
"content": [
{
"type": "output_text",
"text": "Done."
}
]
}
]
}
""")
});
var handler = new RecordingHttpMessageHandler(_ => CreateStreamedTextResponse("Done."));
using var client = CreateClient(handler, reconnectionAttempts: 0);
await client.GetResponseAsync([new ChatMessage(ChatRole.User, "write summary")]);
@@ -205,24 +321,7 @@ public sealed class LiteLlmResponsesChatClientTests
};
}
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("""
{
"output": [
{
"type": "message",
"content": [
{
"type": "output_text",
"text": "Done."
}
]
}
]
}
""")
};
return CreateStreamedTextResponse("Done.");
});
using var client = CreateClient(
handler,
@@ -257,24 +356,7 @@ public sealed class LiteLlmResponsesChatClientTests
};
}
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("""
{
"output": [
{
"type": "message",
"content": [
{
"type": "output_text",
"text": "Done."
}
]
}
]
}
""")
};
return CreateStreamedTextResponse("Done.");
});
using var client = CreateClient(
handler,
@@ -306,7 +388,9 @@ public sealed class LiteLlmResponsesChatClientTests
HttpMessageHandler handler,
int reconnectionAttempts,
LiteLlmResponsesCompactionOptions? compactionOptions = null,
Action? retrying = null)
Action? retrying = null,
Action<string>? reasoningSummaryChanged = null,
bool useStreaming = true)
{
return new LiteLlmResponsesChatClient(
new HttpClient(handler)
@@ -320,7 +404,50 @@ public sealed class LiteLlmResponsesChatClientTests
reconnectionAttempts,
TimeSpan.Zero,
compactionOptions,
retrying: retrying);
retrying: retrying,
reasoningSummaryChanged: reasoningSummaryChanged,
useStreaming: useStreaming);
}
private static HttpResponseMessage CreateStreamedTextResponse(
string text,
params string[] reasoningSummaries)
{
var events = new List<string>();
for (var index = 0; index < reasoningSummaries.Length; index++)
{
events.Add("data: " + JsonSerializer.Serialize(new
{
type = "response.reasoning_summary_text.delta",
item_id = "reasoning_stream",
output_index = 0,
summary_index = index,
delta = reasoningSummaries[index]
}));
}
var outputIndex = reasoningSummaries.Length > 0 ? 1 : 0;
events.Add("data: " + JsonSerializer.Serialize(new
{
type = "response.output_text.delta",
item_id = "msg_stream",
output_index = outputIndex,
content_index = 0,
delta = text
}));
events.Add(
"""data: {"type":"response.completed","response":{"id":"resp_stream","created_at":1779147100,"model":"gpt-5.5","object":"response","output":[],"parallel_tool_calls":true,"status":"completed","store":false,"usage":{"input_tokens":12,"output_tokens":3,"total_tokens":15}}}""");
events.Add("data: [DONE]");
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(
string.Join($"{Environment.NewLine}{Environment.NewLine}", events)
+ Environment.NewLine
+ Environment.NewLine,
Encoding.UTF8,
"text/event-stream")
};
}
private sealed class SequencedHttpMessageHandler : HttpMessageHandler
@@ -13,21 +13,50 @@ namespace MeetingAssistant.Tests;
public sealed class LiteLlmScreenshotOcrClientTests
{
[Fact]
public async Task ExtractUsesAgentStreamingTransportByDefault()
{
var screenshotPath = await CreateScreenshotAsync([1, 2, 3]);
var handler = new RecordingHandler(
CreateStreamedTextResponse("Streamed OCR text"),
"text/event-stream");
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
var options = new MeetingAssistantOptions
{
Agent =
{
Endpoint = "https://summary.local",
Model = "vision-model",
Key = "agent-key",
UseStreaming = true
}
};
var result = await client.ExtractAsync(
screenshotPath,
"Extract screenshot.",
options,
CancellationToken.None);
Assert.Equal("Streamed OCR text", result.Text);
using var payload = JsonDocument.Parse(handler.RequestBody!);
Assert.True(payload.RootElement.GetProperty("stream").GetBoolean());
var message = Assert.Single(payload.RootElement.GetProperty("input").EnumerateArray());
Assert.Equal("message", message.GetProperty("type").GetString());
var content = message.GetProperty("content").EnumerateArray().ToArray();
Assert.Equal("input_text", content[0].GetProperty("type").GetString());
Assert.Equal("Extract screenshot.", content[0].GetProperty("text").GetString());
Assert.Equal("input_image", content[1].GetProperty("type").GetString());
Assert.Equal("data:image/png;base64,AQID", content[1].GetProperty("image_url").GetString());
}
[Fact]
public async Task ExtractUsesAgentEndpointAndModelWhenOcrEndpointAndModelAreBlank()
{
var screenshotPath = await CreateScreenshotAsync([1, 2, 3]);
var handler = new RecordingHandler("""
{
"output": [
{
"content": [
{ "type": "output_text", "text": "Visible slide text" }
]
}
]
}
""");
var handler = new RecordingHandler(CreateNonStreamingTextResponse("Visible slide text"));
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
@@ -37,7 +66,8 @@ public sealed class LiteLlmScreenshotOcrClientTests
{
Endpoint = "https://summary.local",
Model = "summary-model",
Key = "agent-key"
Key = "agent-key",
UseStreaming = false
},
Screenshots =
{
@@ -62,6 +92,7 @@ public sealed class LiteLlmScreenshotOcrClientTests
Assert.Equal("Bearer", handler.Authorization?.Scheme);
Assert.Equal("ocr-key", handler.Authorization?.Parameter);
using var payload = JsonDocument.Parse(handler.RequestBody!);
Assert.False(payload.RootElement.GetProperty("stream").GetBoolean());
Assert.Equal("summary-model", payload.RootElement.GetProperty("model").GetString());
var content = payload.RootElement
.GetProperty("input")[0]
@@ -74,7 +105,7 @@ public sealed class LiteLlmScreenshotOcrClientTests
public async Task ExtractUsesScreenshotOcrEndpointAndModelWhenConfigured()
{
var screenshotPath = await CreateScreenshotAsync([4, 5, 6]);
var handler = new RecordingHandler("""{ "output_text": "OCR result" }""");
var handler = new RecordingHandler(CreateNonStreamingTextResponse("OCR result"));
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
@@ -84,7 +115,8 @@ public sealed class LiteLlmScreenshotOcrClientTests
{
Endpoint = "https://summary.local",
Model = "summary-model",
Key = "agent-key"
Key = "agent-key",
UseStreaming = false
},
Screenshots =
{
@@ -113,11 +145,14 @@ public sealed class LiteLlmScreenshotOcrClientTests
public async Task ExtractParsesCropMetadataAndOmitsMetadataFromReturnedText()
{
var screenshotPath = await CreateScreenshotAsync(CreatePngBytes(8, 6));
var handler = new RecordingHandler("""
{
"output_text": "Slide text\n\n```json\n{ \"crop\": { \"x\": 1, \"y\": 2, \"width\": 3, \"height\": 4 } }\n```"
}
""");
var handler = new RecordingHandler(CreateNonStreamingTextResponse(
"""
Slide text
```json
{ "crop": { "x": 1, "y": 2, "width": 3, "height": 4 } }
```
"""));
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
@@ -125,7 +160,8 @@ public sealed class LiteLlmScreenshotOcrClientTests
{
Agent =
{
Key = "agent-key"
Key = "agent-key",
UseStreaming = false
}
};
@@ -148,11 +184,14 @@ public sealed class LiteLlmScreenshotOcrClientTests
public async Task ExtractParsesAttendeeMetadataAndOmitsMetadataFromReturnedText()
{
var screenshotPath = await CreateScreenshotAsync([1, 2, 3]);
var handler = new RecordingHandler("""
{
"output_text": "Visible participant tiles: Ada and Grace.\n\n```json\n{ \"crop\": null, \"attendees\": [\"Ada Lovelace\", \"Grace Hopper\"] }\n```"
}
""");
var handler = new RecordingHandler(CreateNonStreamingTextResponse(
"""
Visible participant tiles: Ada and Grace.
```json
{ "crop": null, "attendees": ["Ada Lovelace", "Grace Hopper"] }
```
"""));
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
@@ -160,7 +199,8 @@ public sealed class LiteLlmScreenshotOcrClientTests
{
Agent =
{
Key = "agent-key"
Key = "agent-key",
UseStreaming = false
}
};
@@ -178,11 +218,14 @@ public sealed class LiteLlmScreenshotOcrClientTests
public async Task ExtractIgnoresMalformedAttendeesMetadataAndStillParsesCrop()
{
var screenshotPath = await CreateScreenshotAsync(CreatePngBytes(8, 6));
var handler = new RecordingHandler("""
{
"output_text": "Slide text\n\n```json\n{ \"crop\": { \"x\": 1, \"y\": 2, \"width\": 3, \"height\": 4 }, \"attendees\": \"Ada\" }\n```"
}
""");
var handler = new RecordingHandler(CreateNonStreamingTextResponse(
"""
Slide text
```json
{ "crop": { "x": 1, "y": 2, "width": 3, "height": 4 }, "attendees": "Ada" }
```
"""));
var client = new LiteLlmScreenshotOcrClient(
() => handler,
NullLogger<LiteLlmScreenshotOcrClient>.Instance);
@@ -190,7 +233,8 @@ public sealed class LiteLlmScreenshotOcrClientTests
{
Agent =
{
Key = "agent-key"
Key = "agent-key",
UseStreaming = false
}
};
@@ -208,10 +252,12 @@ public sealed class LiteLlmScreenshotOcrClientTests
private sealed class RecordingHandler : HttpMessageHandler
{
private readonly string responseBody;
private readonly string mediaType;
public RecordingHandler(string responseBody)
public RecordingHandler(string responseBody, string mediaType = "application/json")
{
this.responseBody = responseBody;
this.mediaType = mediaType;
}
public Uri? RequestUri { get; private set; }
@@ -231,7 +277,7 @@ public sealed class LiteLlmScreenshotOcrClientTests
: await request.Content.ReadAsStringAsync(cancellationToken);
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent(responseBody, Encoding.UTF8, "application/json")
Content = new StringContent(responseBody, Encoding.UTF8, mediaType)
};
}
}
@@ -249,6 +295,55 @@ public sealed class LiteLlmScreenshotOcrClientTests
return stream.ToArray();
}
private static string CreateNonStreamingTextResponse(string text)
{
return JsonSerializer.Serialize(new
{
id = "resp_ocr",
created_at = 1779147100,
model = "vision-model",
@object = "response",
output = new[]
{
new
{
id = "msg_ocr",
type = "message",
status = "completed",
content = new[]
{
new
{
type = "output_text",
annotations = Array.Empty<object>(),
text
}
},
role = "assistant"
}
},
parallel_tool_calls = true,
status = "completed",
store = false
});
}
private static string CreateStreamedTextResponse(string text)
{
var delta = JsonSerializer.Serialize(new
{
type = "response.output_text.delta",
item_id = "msg_ocr",
output_index = 0,
content_index = 0,
delta = text
});
return
$"data: {delta}{Environment.NewLine}{Environment.NewLine}" +
"""data: {"type":"response.completed","response":{"id":"resp_ocr","created_at":1779147100,"model":"vision-model","object":"response","output":[],"parallel_tool_calls":true,"status":"completed","store":false}}""" +
$"{Environment.NewLine}{Environment.NewLine}data: [DONE]{Environment.NewLine}{Environment.NewLine}";
}
private static async Task<string> CreateScreenshotAsync(byte[] bytes)
{
var screenshotPath = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N") + ".png");
@@ -9,8 +9,8 @@
<ItemGroup>
<PackageReference Include="coverlet.collector" Version="10.0.1" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.9" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.7.0" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.10" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.8.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.5" />
</ItemGroup>
@@ -135,6 +135,72 @@ public sealed class MeetingNoteStoreTests
Assert.Equal("User notes.", loaded.UserNotes);
}
[Fact]
public async Task StoreEscapesApostrophePrefixedAttendeesBeforeWritingFrontmatter()
{
var (store, saved) = await SaveNoteAsync(
title: "Escaped Attendees",
attendees: ["'Ada Lovelace"],
projects: [],
userNotes: "Discuss attendee import.");
var content = await File.ReadAllTextAsync(saved.Path);
var loaded = await store.ReadAsync(saved.Path, CancellationToken.None);
Assert.Contains("- \"'Ada Lovelace\"", content);
Assert.Equal(["'Ada Lovelace"], loaded.Frontmatter.Attendees);
Assert.Equal("Discuss attendee import.", loaded.UserNotes);
}
[Theory]
[InlineData("Ada # platform lead")]
[InlineData("- Ada Lovelace")]
[InlineData("? Ada Lovelace")]
[InlineData("{Ada: Platform}")]
[InlineData("*Ada")]
[InlineData("&Ada")]
[InlineData("!Ada")]
[InlineData("| Ada")]
[InlineData("> Ada")]
[InlineData("@Ada")]
[InlineData("`Ada")]
[InlineData("true")]
[InlineData("null")]
[InlineData("2026-07-08")]
[InlineData("Ada \"The Architect\" Lovelace")]
[InlineData("C:\\People\\Ada")]
public async Task StoreEscapesYamlSensitiveAttendeesBeforeWritingFrontmatter(string attendee)
{
var (store, saved) = await SaveNoteAsync(
title: "Escaped Attendees",
attendees: [attendee],
projects: [],
userNotes: "Discuss attendee import.");
var loaded = await store.ReadAsync(saved.Path, CancellationToken.None);
Assert.Equal([attendee], loaded.Frontmatter.Attendees);
Assert.Equal("Discuss attendee import.", loaded.UserNotes);
}
[Theory]
[InlineData("Planning # Q3")]
[InlineData("- Planning")]
[InlineData("{Planning: Q3}")]
[InlineData("true")]
[InlineData("2026-07-08")]
public async Task StoreEscapesYamlSensitiveScalarAndProjectFrontmatterValues(string value)
{
var (store, saved) = await SaveNoteAsync(
title: value,
attendees: [],
projects: [value],
userNotes: "Discuss YAML escaping.");
var loaded = await store.ReadAsync(saved.Path, CancellationToken.None);
Assert.Equal(value, loaded.Frontmatter.Title);
Assert.Equal([value], loaded.Frontmatter.Projects);
Assert.Equal("Discuss YAML escaping.", loaded.UserNotes);
}
[Fact]
public void ActionLinkEscapesSummaryFileName()
{
@@ -147,6 +213,26 @@ public sealed class MeetingNoteStoreTests
link);
}
private static async Task<(MarkdownMeetingNoteStore Store, MeetingNote Saved)> SaveNoteAsync(
string title,
IReadOnlyList<string> attendees,
IReadOnlyList<string> projects,
string userNotes)
{
var (vaultRoot, store) = CreateStore();
var note = MeetingNoteTemplate.Create(
title: title,
attendees: attendees,
projects: projects,
transcriptPath: Path.Combine(vaultRoot, "Meetings", "Transcripts", "20260519-transcript.md"),
assistantContextPath: Path.Combine(vaultRoot, "Meetings", "Assistant Context", "20260519-context.md"),
summaryPath: Path.Combine(vaultRoot, "Meetings", "Summaries", "20260519-summary.md"),
userNotes: userNotes);
var saved = await store.SaveAsync(note, CancellationToken.None);
return (store, saved);
}
private static (string VaultRoot, MarkdownMeetingNoteStore Store) CreateStore()
{
var vaultRoot = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
@@ -1,6 +1,7 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Screenshots;
using MeetingAssistant.Speakers;
using MeetingAssistant.Workflow;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using System.Drawing;
@@ -167,6 +168,35 @@ public sealed class MeetingScreenshotServiceTests
request.SequenceEqual(["Ada Lovelace", "Ada L.", "Grace Hopper", "Ada Lovelace"]));
}
[Fact]
public async Task CaptureTransformsOcrAttendeesBeforeWritingMeetingNote()
{
var fixture = await ScreenshotFixture.CreateAsync(
options =>
{
options.Screenshots.Ocr.Enabled = true;
});
var workflowEngine = new TransformingAttendeeWorkflowEngine("Ada Lovelace (Contoso)", "Ada Lovelace");
var service = fixture.CreateService(
new FixedScreenshotCapture([1, 2, 3]),
new CapturingScreenshotOcrClient(
"Visible participant tile: Ada.",
attendees: ["Ada Lovelace (Contoso)"]),
meetingWorkflowEngine: workflowEngine);
await service.CaptureAsync(
fixture.Artifacts,
DateTimeOffset.Parse("2026-05-26T10:00:00+02:00"),
DateTimeOffset.Parse("2026-05-26T10:00:10+02:00"),
fixture.Options,
CancellationToken.None);
await service.WaitForPendingOcrAsync(fixture.Artifacts, TimeSpan.FromSeconds(5), CancellationToken.None);
var meeting = await fixture.NoteStore.ReadAsync(fixture.Artifacts.MeetingNotePath, CancellationToken.None);
Assert.Equal(["Ada Lovelace"], meeting.Frontmatter.Attendees);
Assert.Equal(["Ada Lovelace (Contoso)"], workflowEngine.AttendeeRequests);
}
[Fact]
public async Task CaptureWritesRetryLinkWhenOcrFails()
{
@@ -423,7 +453,8 @@ public sealed class MeetingScreenshotServiceTests
public MeetingScreenshotService CreateService(
IActiveWindowScreenshotCapture capture,
IScreenshotOcrClient ocrClient,
ISpeakerIdentityAttendeeCanonicalizer? attendeeCanonicalizer = null)
ISpeakerIdentityAttendeeCanonicalizer? attendeeCanonicalizer = null,
IMeetingWorkflowEngine? meetingWorkflowEngine = null)
{
return new MeetingScreenshotService(
capture,
@@ -431,7 +462,8 @@ public sealed class MeetingScreenshotServiceTests
NoteStore,
attendeeCanonicalizer ?? PassthroughSpeakerIdentityAttendeeCanonicalizer.Instance,
ocrClient,
NullLogger<MeetingScreenshotService>.Instance);
NullLogger<MeetingScreenshotService>.Instance,
meetingWorkflowEngine);
}
}
@@ -24,6 +24,7 @@ public sealed class MeetingSummaryInstructionBuilderTests
Assert.Contains("You are the Meeting Assistant summary agent.", instructions);
Assert.Contains("include only the most relevant cropped screenshots", instructions);
Assert.Contains("Do not include every cropped screenshot", instructions);
Assert.Contains("encode spaces in image-link targets as `%20`", instructions);
Assert.Contains("Use add_attendee and remove_attendee", instructions);
Assert.Contains("partial screenshot", instructions);
Assert.Contains("override_speaker", instructions);
@@ -39,6 +40,27 @@ public sealed class MeetingSummaryInstructionBuilderTests
Assert.Contains("title parameter", instructions);
}
[Fact]
public async Task DefaultPromptTreatsAssistantContextAsMeetingMemoryForUncertainty()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var artifacts = CreateArtifacts(root);
await WriteMeetingNoteAsync(artifacts.MeetingNotePath, []);
var builder = new MeetingSummaryInstructionBuilder(Options.Create(new MeetingAssistantOptions
{
Agent = new AgentOptions { InitialPrompt = " " },
Vault = new VaultOptions { ProjectsFolder = Path.Combine(root, "Projects") }
}));
var instructions = await builder.BuildAsync(artifacts, CancellationToken.None);
Assert.Contains("meeting-specific memory", instructions);
Assert.Contains("unexpected problems", instructions);
Assert.Contains("missing information", instructions);
Assert.Contains("assumptions", instructions);
Assert.Contains("write_context", instructions);
}
[Fact]
public async Task BuilderUsesConfiguredPrompt()
{
@@ -293,6 +293,44 @@ public sealed class MeetingSummaryToolTests
Assert.Contains("transcript: \"[[../Transcripts/transcript|Transcript]]\"", meetingNote);
}
[Fact]
public async Task ToolsTransformAddedAttendeeBeforeWritingMeetingNote()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var artifacts = new MeetingSessionArtifacts(
MeetingNotePath: Path.Combine(root, "Meetings", "Notes", "meeting.md"),
TranscriptPath: Path.Combine(root, "Meetings", "Transcripts", "transcript.md"),
AssistantContextPath: Path.Combine(root, "Meetings", "Assistant Context", "context.md"),
SummaryPath: Path.Combine(root, "Meetings", "Summaries", "summary.md"));
Directory.CreateDirectory(Path.GetDirectoryName(artifacts.MeetingNotePath)!);
await File.WriteAllTextAsync(
artifacts.MeetingNotePath,
"""
---
title: Meeting
attendees: []
projects: []
transcript: "[[../Transcripts/transcript|Transcript]]"
assistant_context: "[[../Assistant Context/context|Assistant Context]]"
summary: "[[../Summaries/summary|Summary]]"
---
User note line.
""");
var workflowEngine = new TransformingAttendeeWorkflowEngine("Ada Lovelace (Contoso)", "Ada Lovelace");
var tools = new MeetingSummaryTools(
artifacts,
new MeetingAssistantOptions(),
meetingWorkflowEngine: workflowEngine);
Assert.Equal("Added attendee Ada Lovelace.", await tools.AddAttendee("Ada Lovelace (Contoso)"));
var meetingNote = await File.ReadAllTextAsync(artifacts.MeetingNotePath);
Assert.Contains("- Ada Lovelace", meetingNote);
Assert.DoesNotContain("Ada Lovelace (Contoso)", meetingNote);
Assert.Equal(["Ada Lovelace (Contoso)"], workflowEngine.AttendeeRequests);
}
[Fact]
public async Task ToolsOverrideSpeakerInTranscriptAndRecordOverride()
{
@@ -799,4 +837,5 @@ public sealed class MeetingSummaryToolTests
Assert.False(tools.SummaryWasWritten);
Assert.False(File.Exists(artifacts.SummaryPath));
}
}
@@ -310,6 +310,10 @@ public sealed class MeetingWorkflowEngineTests
[InlineData("speaker_identified:\n name: ADA", "speaker", null, null, "ada", true)]
[InlineData("speaker_identified:\n name: Ada", "speaker", null, null, "Grace", false)]
[InlineData("speaker_identified: {}", "speaker", null, null, "Grace", true)]
[InlineData("attendee_added:\n equals: Ada Lovelace", "attendee", null, null, "ada lovelace", true)]
[InlineData("attendee_added:\n contains: contoso", "attendee", null, null, "Ada (Contoso)", true)]
[InlineData("attendee_added:\n regex: '^Ada .+Contoso\\)$'", "attendee", null, null, "Ada (Contoso)", true)]
[InlineData("attendee_added:\n regex: '^Grace'", "attendee", null, null, "Ada (Contoso)", false)]
public async Task TriggerMatchingScenarios(
string triggerYaml,
string eventKind,
@@ -318,6 +322,16 @@ public sealed class MeetingWorkflowEngineTests
string? speaker,
bool shouldRun)
{
var stepYaml = eventKind == "attendee"
? """
- uses: set_property
property: attendee.name
value: Triggered
"""
: """
- uses: add_project
value: Triggered
""";
var fixture = await WorkflowFixture.CreateAsync(
$$"""
rules:
@@ -325,14 +339,21 @@ public sealed class MeetingWorkflowEngineTests
on:
- {{triggerYaml}}
steps:
- uses: add_project
value: Triggered
{{stepYaml}}
""");
await fixture.Engine.RunAsync(
CreateEvent(eventKind, fixture.Artifacts, from, to, speaker),
fixture.Options,
CancellationToken.None);
var workflowEvent = CreateEvent(eventKind, fixture.Artifacts, from, to, speaker);
if (eventKind == "attendee")
{
var transformed = await fixture.Engine.TransformAttendeeAsync(
workflowEvent,
fixture.Options,
CancellationToken.None);
Assert.Equal(shouldRun ? "Triggered" : speaker, transformed);
return;
}
await fixture.Engine.RunAsync(workflowEvent, fixture.Options, CancellationToken.None);
var meeting = await fixture.ReadMeetingAsync();
Assert.Equal(shouldRun, meeting.Frontmatter.Projects.Contains("Triggered"));
@@ -603,6 +624,34 @@ public sealed class MeetingWorkflowEngineTests
Assert.Equal(["Ada Lovelace"], meeting.Frontmatter.Attendees);
}
[Fact]
public async Task AttendeeAddedRuleCanTransformWorkflowAddedAttendee()
{
var fixture = await WorkflowFixture.CreateAsync(
"""
rules:
- name: add-raw-attendee
on:
- created: {}
steps:
- uses: add_attendee
value: 'Ada Lovelace (Contoso)'
- name: clean-contoso-attendee
on:
- attendee_added:
contains: 'Contoso'
steps:
- uses: set_property
property: attendee.name
value: '@Model.Attendee.Name.Replace(" (Contoso)", "")'
""");
await RunCreatedAsync(fixture);
var meeting = await fixture.ReadMeetingAsync();
Assert.Equal(["Ada Lovelace"], meeting.Frontmatter.Attendees);
}
[Fact]
public async Task AddAttendeeDoesNotDuplicateExistingDisplayNameFromEmailAddress()
{
@@ -792,6 +841,7 @@ public sealed class MeetingWorkflowEngineTests
{
"created" => MeetingWorkflowEvent.Created(artifacts),
"speaker" => MeetingWorkflowEvent.SpeakerIdentified(artifacts, speaker ?? "Ada"),
"attendee" => MeetingWorkflowEvent.AttendeeAdded(artifacts, speaker ?? "Ada"),
"state" => MeetingWorkflowEvent.StateTransition(
artifacts,
ParseState(from ?? "collecting metadata"),
@@ -0,0 +1,109 @@
using MeetingAssistant.Recording;
using Microsoft.Extensions.Logging.Abstractions;
namespace MeetingAssistant.Tests;
public sealed class MicrophoneAudioSourceTests
{
[Fact]
public async Task CaptureMovesToNewlyResolvedMicrophoneWhenCurrentCaptureFails()
{
var captureSources = new SequenceMicrophoneCaptureSourceFactory(
new FailingAfterChunkAudioSource(Pcm16(1_000)),
new ActiveAudioSource(Pcm16(2_000)));
var source = new MicrophoneAudioSource(
captureSources,
NullLogger<MicrophoneAudioSource>.Instance,
TimeSpan.Zero);
using var cancellation = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await using var chunks = source
.CaptureAsync(new MeetingAssistantOptions(), cancellation.Token)
.GetAsyncEnumerator(cancellation.Token);
Assert.True(await chunks.MoveNextAsync());
Assert.Equal(1_000, BitConverter.ToInt16(chunks.Current.Pcm));
Assert.True(await chunks.MoveNextAsync());
Assert.Equal(2_000, BitConverter.ToInt16(chunks.Current.Pcm));
Assert.Equal(2, captureSources.CaptureCreationCount);
await cancellation.CancelAsync();
}
[Fact]
public async Task CaptureWaitsForMicrophoneToBecomeAvailable()
{
var captureSources = new InitiallyUnavailableMicrophoneCaptureSourceFactory(
new ActiveAudioSource(Pcm16(3_000)));
var source = new MicrophoneAudioSource(
captureSources,
NullLogger<MicrophoneAudioSource>.Instance,
TimeSpan.Zero);
using var cancellation = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await using var chunks = source
.CaptureAsync(new MeetingAssistantOptions(), cancellation.Token)
.GetAsyncEnumerator(cancellation.Token);
Assert.True(await chunks.MoveNextAsync());
Assert.Equal(3_000, BitConverter.ToInt16(chunks.Current.Pcm));
Assert.Equal(2, captureSources.CaptureCreationCount);
await cancellation.CancelAsync();
}
private static byte[] Pcm16(short sample)
{
return BitConverter.GetBytes(sample);
}
private sealed class SequenceMicrophoneCaptureSourceFactory(params IMeetingAudioSource[] sources)
: IMicrophoneCaptureSourceFactory
{
private readonly Queue<IMeetingAudioSource> sources = new(sources);
public int CaptureCreationCount { get; private set; }
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options)
{
CaptureCreationCount++;
return sources.Dequeue();
}
}
private sealed class InitiallyUnavailableMicrophoneCaptureSourceFactory(IMeetingAudioSource availableSource)
: IMicrophoneCaptureSourceFactory
{
public int CaptureCreationCount { get; private set; }
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options)
{
CaptureCreationCount++;
if (CaptureCreationCount == 1)
{
throw new InvalidOperationException("No microphone is currently available.");
}
return availableSource;
}
}
private sealed class FailingAfterChunkAudioSource(byte[] pcm) : IMeetingAudioSource
{
public async IAsyncEnumerable<AudioChunk> CaptureAsync(
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
await Task.Yield();
yield return new AudioChunk(pcm, 16000, 1);
throw new InvalidOperationException("The active microphone was disconnected.");
}
}
private sealed class ActiveAudioSource(byte[] pcm) : IMeetingAudioSource
{
public async IAsyncEnumerable<AudioChunk> CaptureAsync(
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
yield return new AudioChunk(pcm, 16000, 1);
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
}
}
}
@@ -51,4 +51,17 @@ public sealed class MicrophoneSelectionTests
Assert.Equal("runtime-id", selected?.Id);
}
[Fact]
public void UnavailableDefaultMicrophoneFallsBackToAnotherActiveDevice()
{
var selection = new MicrophoneDeviceSelection();
var selected = selection.Resolve(
configuredDeviceId: null,
new MicrophoneDevice("disconnected-id", "disconnected microphone"),
[new MicrophoneDevice("backup-id", "backup microphone")]);
Assert.Equal("backup-id", selected?.Id);
}
}
@@ -40,7 +40,7 @@ public sealed class OutlookMeetingCandidateSelectorTests
{
var now = new DateTime(2026, 5, 20, 10, 0, 0);
var endingOverlap = new Candidate(now.AddMinutes(-25), now.AddMinutes(2));
var upcoming = new Candidate(now.AddMinutes(5), now.AddMinutes(35));
var upcoming = new Candidate(now.AddMinutes(1), now.AddMinutes(31));
var selected = OutlookMeetingCandidateSelector.Select(
[endingOverlap, upcoming],
@@ -117,7 +117,7 @@ public sealed class RecordingCoordinatorTests
}
[Fact]
public async Task TranscriptLineWorkflowRuleTransformsLiveTranscriptBeforePersistingMarkdown()
public async Task TranscriptLineWorkflowRuleTransformsLiveTranscriptAfterDurableAppend()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var rulesPath = Path.Combine(root, "rules.yaml");
@@ -900,6 +900,218 @@ public sealed class RecordingCoordinatorTests
await coordinator.StopAsync(CancellationToken.None);
}
[Fact]
public async Task AttachPromptedMetadataUpdatesTheActiveMeetingWithoutInterruptingRecording()
{
var audioSource = new ControlledAudioSource();
var transcriptStore = new InMemoryTranscriptStore();
var noteStore = new InMemoryMeetingNoteStore("C:\\Vault\\Meetings\\Notes\\active-metadata-meeting.md");
var artifactStore = new InMemoryMeetingArtifactStore();
var workflowEngine = new TransformingAttendeeWorkflowEngine(
"Ada Lovelace (Contoso)",
"Ada Lovelace");
var coordinator = new MeetingRecordingCoordinator(
audioSource,
new TestSpeechRecognitionPipelineFactory(new EchoStreamingTranscriptionProvider()),
transcriptStore,
noteStore,
new CapturingMeetingNoteOpener(),
artifactStore,
new InMemoryRecordedAudioStore(),
new CapturingMeetingSummaryPipeline(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<MeetingRecordingCoordinator>.Instance,
meetingMetadataProvider: new CountingMeetingMetadataProvider(),
meetingWorkflowEngine: workflowEngine);
var started = await coordinator.StartAsync(CancellationToken.None);
noteStore.UpdateSavedNote(noteStore.SavedNote! with { UserNotes = "Keep this user note." });
var promptedMetadata = new MeetingMetadata(
"Selected architecture sync",
["Ada Lovelace (Contoso)"],
"Review the selected architecture",
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00"));
var attached = await coordinator.AttachMetadataToCurrentMeetingAsync(
promptedMetadata,
CancellationToken.None);
Assert.True(started.IsRecording);
Assert.True(attached.IsRecording);
Assert.Equal(started.TranscriptPath, attached.TranscriptPath);
Assert.Equal("Selected architecture sync", noteStore.SavedNote?.Frontmatter.Title);
Assert.Equal(["Ada Lovelace"], noteStore.SavedNote?.Frontmatter.Attendees);
Assert.Equal("Keep this user note.", noteStore.SavedNote?.UserNotes);
Assert.Equal("Selected architecture sync", artifactStore.ContextMeetingNote?.Frontmatter.Title);
Assert.Equal("Review the selected architecture", artifactStore.Agenda);
Assert.Equal(
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00"),
artifactStore.ScheduledEnd);
Assert.Equal(2, transcriptStore.MetadataUpdateCount);
Assert.Equal("Selected architecture sync", transcriptStore.MetadataMeetingNote?.Frontmatter.Title);
await coordinator.StopAsync(CancellationToken.None);
}
[Fact]
public async Task AttachedPromptMetadataWinsOverAStandaloneLookupThatCompletesLater()
{
var audioSource = new ControlledAudioSource();
var noteStore = new InMemoryMeetingNoteStore("C:\\Vault\\Meetings\\Notes\\explicit-metadata-meeting.md");
var artifactStore = new InMemoryMeetingArtifactStore();
var metadataProvider = new BlockingMeetingMetadataProvider(new MeetingMetadata(
"Background calendar match",
["Grace"],
"Background agenda",
DateTimeOffset.Parse("2026-05-19T12:00:00+02:00")));
var coordinator = new MeetingRecordingCoordinator(
audioSource,
new TestSpeechRecognitionPipelineFactory(new EchoStreamingTranscriptionProvider()),
new InMemoryTranscriptStore(),
noteStore,
new CapturingMeetingNoteOpener(),
artifactStore,
new InMemoryRecordedAudioStore(),
new CapturingMeetingSummaryPipeline(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<MeetingRecordingCoordinator>.Instance,
meetingMetadataProvider: metadataProvider);
var promptedMetadata = new MeetingMetadata(
"Explicit prompted appointment",
["Ada"],
"Explicit agenda",
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00"));
await coordinator.StartAsync(CancellationToken.None);
await metadataProvider.WaitUntilRequestedAsync();
await coordinator.AttachMetadataToCurrentMeetingAsync(promptedMetadata, CancellationToken.None);
metadataProvider.Release();
await WaitUntilAsync(() => artifactStore.States.Contains(AssistantContextState.Transcribing));
Assert.Equal("Explicit prompted appointment", noteStore.SavedNote?.Frontmatter.Title);
Assert.Equal(["Ada"], noteStore.SavedNote?.Frontmatter.Attendees);
Assert.Equal("Explicit agenda", artifactStore.Agenda);
Assert.Equal(
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00"),
artifactStore.ScheduledEnd);
await coordinator.StopAsync(CancellationToken.None);
}
[Fact]
public async Task AttachPromptedMetadataDoesNothingAfterTheActiveRecordingStops()
{
var audioSource = new ControlledAudioSource();
var transcriptStore = new InMemoryTranscriptStore();
var noteStore = new InMemoryMeetingNoteStore("C:\\Vault\\Meetings\\Notes\\stopped-metadata-meeting.md");
var coordinator = new MeetingRecordingCoordinator(
audioSource,
new TestSpeechRecognitionPipelineFactory(new EchoStreamingTranscriptionProvider()),
transcriptStore,
noteStore,
new CapturingMeetingNoteOpener(),
new InMemoryMeetingArtifactStore(),
new InMemoryRecordedAudioStore(),
new CapturingMeetingSummaryPipeline(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<MeetingRecordingCoordinator>.Instance,
meetingMetadataProvider: new CountingMeetingMetadataProvider());
await coordinator.StartAsync(CancellationToken.None);
var stopped = await coordinator.StopAsync(CancellationToken.None);
var metadataUpdatesBeforeAttach = transcriptStore.MetadataUpdateCount;
var titleBeforeAttach = noteStore.SavedNote?.Frontmatter.Title;
var result = await coordinator.AttachMetadataToCurrentMeetingAsync(
new MeetingMetadata(
"Stale prompted appointment",
["Ada"],
"Stale agenda",
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00")),
CancellationToken.None);
Assert.False(stopped.IsRecording);
Assert.False(result.IsRecording);
Assert.Equal(titleBeforeAttach, noteStore.SavedNote?.Frontmatter.Title);
Assert.Equal(metadataUpdatesBeforeAttach, transcriptStore.MetadataUpdateCount);
}
[Fact]
public async Task FailedPromptMetadataAttachmentDoesNotSuppressTheBackgroundLookup()
{
var audioSource = new ControlledAudioSource();
var noteStore = new InMemoryMeetingNoteStore("C:\\Vault\\Meetings\\Notes\\failed-attach-metadata-meeting.md");
var artifactStore = new InMemoryMeetingArtifactStore(failFirstMetadataUpdate: true);
var metadataProvider = new BlockingMeetingMetadataProvider(new MeetingMetadata(
"Background calendar fallback",
["Grace"],
"Background fallback agenda",
DateTimeOffset.Parse("2026-05-19T12:00:00+02:00")));
var coordinator = new MeetingRecordingCoordinator(
audioSource,
new TestSpeechRecognitionPipelineFactory(new EchoStreamingTranscriptionProvider()),
new InMemoryTranscriptStore(),
noteStore,
new CapturingMeetingNoteOpener(),
artifactStore,
new InMemoryRecordedAudioStore(),
new CapturingMeetingSummaryPipeline(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<MeetingRecordingCoordinator>.Instance,
meetingMetadataProvider: metadataProvider);
await coordinator.StartAsync(CancellationToken.None);
await metadataProvider.WaitUntilRequestedAsync();
await Assert.ThrowsAsync<InvalidOperationException>(() =>
coordinator.AttachMetadataToCurrentMeetingAsync(
new MeetingMetadata(
"Prompt attachment that fails",
["Ada"],
"Prompt agenda",
DateTimeOffset.Parse("2026-05-19T11:00:00+02:00")),
CancellationToken.None));
metadataProvider.Release();
await WaitUntilAsync(() => artifactStore.States.Contains(AssistantContextState.Transcribing));
Assert.Equal("Background calendar fallback", noteStore.SavedNote?.Frontmatter.Title);
Assert.Equal(["Grace"], noteStore.SavedNote?.Frontmatter.Attendees);
Assert.Equal("Background fallback agenda", artifactStore.Agenda);
await coordinator.StopAsync(CancellationToken.None);
}
[Fact]
public async Task StartTransformsMetadataAttendeesBeforeWritingNote()
{
var audioSource = new ControlledAudioSource();
var noteStore = new InMemoryMeetingNoteStore("C:\\Vault\\Meetings\\Notes\\metadata-meeting.md");
var workflowEngine = new TransformingAttendeeWorkflowEngine("Ada Lovelace (Contoso)", "Ada Lovelace");
var coordinator = new MeetingRecordingCoordinator(
audioSource,
new TestSpeechRecognitionPipelineFactory(new EchoStreamingTranscriptionProvider()),
new InMemoryTranscriptStore(),
noteStore,
new CapturingMeetingNoteOpener(),
new InMemoryMeetingArtifactStore(),
new InMemoryRecordedAudioStore(),
new CapturingMeetingSummaryPipeline(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<MeetingRecordingCoordinator>.Instance,
meetingMetadataProvider: new FixedMeetingMetadataProvider(new MeetingMetadata(
"Architecture Sync",
["Ada Lovelace (Contoso)"],
"",
null)),
meetingWorkflowEngine: workflowEngine);
await coordinator.StartAsync(CancellationToken.None);
await WaitUntilAsync(() => noteStore.SavedNote?.Frontmatter.Attendees.Count > 0);
Assert.Equal(["Ada Lovelace"], noteStore.SavedNote?.Frontmatter.Attendees);
Assert.Equal(["Ada Lovelace (Contoso)"], workflowEngine.AttendeeRequests);
await coordinator.StopAsync(CancellationToken.None);
}
[Fact]
public async Task StartSkipsOutlookMeetingAttendeesAboveConfiguredImportLimit()
{
@@ -2792,6 +3004,8 @@ public sealed class RecordingCoordinatorTests
public MeetingNote? MetadataMeetingNote { get; private set; }
public int MetadataUpdateCount { get; private set; }
public Task ReplaceLinesAsync(
TranscriptSession session,
IReadOnlyList<string> replacementLines,
@@ -2807,6 +3021,7 @@ public sealed class RecordingCoordinatorTests
MeetingNote meetingNote,
CancellationToken cancellationToken)
{
MetadataUpdateCount++;
MetadataMeetingNote = meetingNote;
return Task.CompletedTask;
}
@@ -3233,10 +3448,14 @@ public sealed class RecordingCoordinatorTests
private sealed class InMemoryMeetingArtifactStore : IMeetingArtifactStore
{
private readonly bool createAssistantContextFile;
private bool failNextMetadataUpdate;
public InMemoryMeetingArtifactStore(bool createAssistantContextFile = false)
public InMemoryMeetingArtifactStore(
bool createAssistantContextFile = false,
bool failFirstMetadataUpdate = false)
{
this.createAssistantContextFile = createAssistantContextFile;
failNextMetadataUpdate = failFirstMetadataUpdate;
}
public MeetingSessionArtifacts? CreatedArtifacts { get; private set; }
@@ -3294,6 +3513,12 @@ public sealed class RecordingCoordinatorTests
DateTimeOffset? scheduledEnd,
CancellationToken cancellationToken)
{
if (failNextMetadataUpdate)
{
failNextMetadataUpdate = false;
throw new InvalidOperationException("Metadata artifact update failed.");
}
ContextMeetingNote = meetingNote;
Agenda = agenda;
ScheduledEnd = scheduledEnd;
@@ -3398,6 +3623,7 @@ public sealed class RecordingCoordinatorTests
private sealed class BlockingMeetingMetadataProvider : IMeetingMetadataProvider
{
private readonly MeetingMetadata metadata;
private readonly TaskCompletionSource requested = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource release = new(TaskCreationOptions.RunContinuationsAsynchronously);
public BlockingMeetingMetadataProvider(MeetingMetadata metadata)
@@ -3410,10 +3636,16 @@ public sealed class RecordingCoordinatorTests
release.TrySetResult();
}
public Task WaitUntilRequestedAsync()
{
return requested.Task.WaitAsync(TimeSpan.FromSeconds(5));
}
public async Task<MeetingMetadata?> GetCurrentMeetingAsync(
DateTimeOffset startedAt,
CancellationToken cancellationToken)
{
requested.TrySetResult();
await release.Task.WaitAsync(cancellationToken);
return metadata;
}
+54 -30
View File
@@ -13,22 +13,23 @@ public sealed class TaskbarIconTests
{
var menu = MeetingTaskbarMenuBuilder.Build(
Status(),
[Profile("default", "Ctrl+Alt+M"), Profile("english", "Ctrl+Alt+L")]);
[Profile("default", "Ctrl+Alt+M"), Profile("english", "Ctrl+Alt+L")],
[new MicrophoneDevice("integrated", "integrated microphone")],
"integrated");
Assert.Equal(RecordingProcessState.Idle, menu.State);
Assert.Contains(menu.Items, item =>
item.Action == MeetingTaskbarAction.EditRules &&
item.Text == "Settings and logs");
Assert.Contains(menu.Items, item =>
item.Action == MeetingTaskbarAction.StartRecording &&
item.ProfileName == "default" &&
item.Text == "Start meeting recording (default)\tCtrl+Alt+M");
Assert.Contains(menu.Items, item =>
item.Action == MeetingTaskbarAction.StartRecording &&
item.ProfileName == "english" &&
item.Text == "Start meeting recording (english)\tCtrl+Alt+L");
Assert.DoesNotContain(menu.Items, item => item.Action == MeetingTaskbarAction.StopRecording);
Assert.DoesNotContain(menu.Items, item => item.Action == MeetingTaskbarAction.AbortRecording);
AssertMenuLayout(
menu,
("Open agent", MeetingTaskbarAction.EditRules, false),
("Microphone", MeetingTaskbarAction.OpenSubmenu, true),
("Start meeting recording (default)\tCtrl+Alt+M", MeetingTaskbarAction.StartRecording, false),
("Start meeting recording (english)\tCtrl+Alt+L", MeetingTaskbarAction.StartRecording, false),
("Exit", MeetingTaskbarAction.Exit, true));
Assert.Equal(
["default", "english"],
menu.Items
.Where(item => item.Action == MeetingTaskbarAction.StartRecording)
.Select(item => item.ProfileName));
}
[Fact]
@@ -60,27 +61,41 @@ public sealed class TaskbarIconTests
}
[Fact]
public void RecordingMenuOffersStopAbortAndOtherProfileSwitches()
public void RecordingMenuPrioritizesFinishMeetingInDedicatedSection()
{
var menu = MeetingTaskbarMenuBuilder.Build(
Status(isRecording: true, state: RecordingProcessState.Recording, profile: "default"),
[Profile("default", "Ctrl+Alt+M"), Profile("english", "Ctrl+Alt+L"), Profile("french", "Ctrl+Alt+F")]);
[Profile("default", "Ctrl+Alt+M"), Profile("english", "Ctrl+Alt+L")],
[new MicrophoneDevice("integrated", "integrated microphone")],
"integrated");
Assert.Equal(RecordingProcessState.Recording, menu.State);
Assert.Contains(menu.Items, item => item.Action == MeetingTaskbarAction.StopRecording);
Assert.Contains(menu.Items, item => item.Action == MeetingTaskbarAction.AbortRecording);
Assert.Contains(menu.Items, item =>
item.Action == MeetingTaskbarAction.SwitchProfile &&
item.ProfileName == "english" &&
item.Text == "Switch to english\tCtrl+Alt+L");
Assert.Contains(menu.Items, item =>
item.Action == MeetingTaskbarAction.SwitchProfile &&
item.ProfileName == "french" &&
item.Text == "Switch to french\tCtrl+Alt+F");
Assert.DoesNotContain(menu.Items, item =>
item.Action == MeetingTaskbarAction.SwitchProfile &&
item.ProfileName == "default");
Assert.DoesNotContain(menu.Items, item => item.Action == MeetingTaskbarAction.StartRecording);
AssertMenuLayout(
menu,
("Open agent", MeetingTaskbarAction.EditRules, false),
("Finish meeting", MeetingTaskbarAction.StopRecording, true),
("Microphone", MeetingTaskbarAction.OpenSubmenu, true),
("Cancel meeting recording and discard", MeetingTaskbarAction.AbortRecording, false),
("Switch to english\tCtrl+Alt+L", MeetingTaskbarAction.SwitchProfile, false),
("Exit", MeetingTaskbarAction.Exit, true));
Assert.Equal(
"english",
Assert.Single(menu.Items, item => item.Action == MeetingTaskbarAction.SwitchProfile).ProfileName);
}
[Fact]
public void RecordingMenuKeepsFinishMeetingIsolatedWithoutMicrophones()
{
var menu = MeetingTaskbarMenuBuilder.Build(
Status(isRecording: true, state: RecordingProcessState.Recording, profile: "default"),
[Profile("default")]);
AssertMenuLayout(
menu,
("Open agent", MeetingTaskbarAction.EditRules, false),
("Finish meeting", MeetingTaskbarAction.StopRecording, true),
("Cancel meeting recording and discard", MeetingTaskbarAction.AbortRecording, true),
("Exit", MeetingTaskbarAction.Exit, true));
}
[Fact]
@@ -173,6 +188,15 @@ public sealed class TaskbarIconTests
});
}
private static void AssertMenuLayout(
MeetingTaskbarMenu menu,
params (string Text, MeetingTaskbarAction Action, bool StartsSection)[] expected)
{
Assert.Equal(
expected,
menu.Items.Select(item => (item.Text, item.Action, item.StartsSection)));
}
private static RecordingStatus Status(
bool isRecording = false,
RecordingProcessState state = RecordingProcessState.Idle,
@@ -0,0 +1,44 @@
using MeetingAssistant.Workflow;
namespace MeetingAssistant.Tests;
internal sealed class TransformingAttendeeWorkflowEngine : IMeetingWorkflowEngine
{
private readonly string source;
private readonly string replacement;
public TransformingAttendeeWorkflowEngine(string source, string replacement)
{
this.source = source;
this.replacement = replacement;
}
public List<string> AttendeeRequests { get; } = [];
public Task RunAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
public Task<string> TransformTranscriptLineAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return Task.FromResult(workflowEvent.TranscriptLineText ?? "");
}
public Task<string> TransformAttendeeAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
AttendeeRequests.Add(workflowEvent.AttendeeName ?? "");
return Task.FromResult(string.Equals(workflowEvent.AttendeeName, source, StringComparison.OrdinalIgnoreCase)
? replacement
: workflowEvent.AttendeeName ?? "");
}
}
@@ -20,6 +20,7 @@ public sealed class WorkflowRulesEditorTests
Key = "summary-key",
KeyEnv = "SUMMARY_KEY",
Model = "summary-model",
UseStreaming = true,
EnableThinking = true,
ReasoningEffort = ReasoningEffortOption.High,
ReconnectionAttempts = 7,
@@ -33,6 +34,7 @@ public sealed class WorkflowRulesEditorTests
var editor = new WorkflowRulesEditorOptions
{
Model = "editor-model",
UseStreaming = false,
EnableThinking = false,
MaxOutputTokens = 50
};
@@ -43,6 +45,7 @@ public sealed class WorkflowRulesEditorTests
Assert.Equal("summary-key", effective.Key);
Assert.Equal("SUMMARY_KEY", effective.KeyEnv);
Assert.Equal("editor-model", effective.Model);
Assert.False(effective.UseStreaming);
Assert.False(effective.EnableThinking);
Assert.Equal(ReasoningEffortOption.High, effective.ReasoningEffort);
Assert.Equal(7, effective.ReconnectionAttempts);
@@ -637,6 +640,28 @@ public sealed class WorkflowRulesEditorTests
Assert.Equal("rules: []", await File.ReadAllTextAsync(fixture.RulesPath));
}
[Fact]
public async Task RulesEditorToolsRefuseSideEffectingAttendeeAddedRules()
{
var fixture = await CreateRulesEditorFixtureAsync();
var result = await fixture.Tools.WriteRules("""
rules:
- name: attendee-side-effect
on:
- attendee_added:
contains: Contoso
steps:
- uses: add_context
value: '@Model.Attendee.Name'
""", replace_file: true);
Assert.StartsWith("Refused: workflow rules are invalid.", result);
Assert.Contains("attendee-side-effect", result);
Assert.Contains("attendee.name", result);
Assert.Equal("rules: []", await File.ReadAllTextAsync(fixture.RulesPath));
}
[Fact]
public async Task RulesEditorToolsAcceptAndParseMaskedProfanityRedactionRule()
{
@@ -738,6 +763,29 @@ public sealed class WorkflowRulesEditorTests
Assert.Contains("matching correction", instructions);
}
[Fact]
public async Task InstructionBuilderTreatsAssistantContextAsMeetingMemoryDuringRepairs()
{
var builder = new WorkflowRulesEditorInstructionBuilder(
NullLogger<WorkflowRulesEditorInstructionBuilder>.Instance);
var options = new MeetingAssistantOptions
{
WorkflowRulesEditor = new WorkflowRulesEditorOptions
{
InitialPrompt = "Custom interactive agent instructions."
}
};
var instructions = await builder.BuildAsync(options, CancellationToken.None);
Assert.Contains("Custom interactive agent instructions.", instructions);
Assert.Contains("meeting-specific memory", instructions);
Assert.Contains("fix or investigate a meeting or summary", instructions);
Assert.Contains("read the matching assistant context", instructions);
Assert.Contains("problems, missing information, assumptions, prior fixes, and conclusions", instructions);
Assert.Contains("append a concise record of your fixes and conclusions", instructions);
}
[Fact]
public async Task RulesEditorToolsCrudAndSearchSpeakerIdentities()
{
@@ -857,15 +905,13 @@ public sealed class WorkflowRulesEditorTests
Assert.Equal("", viewModel.Draft);
Assert.Equal(
[new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, "Rename ABS Daily")],
viewModel.Messages.ToArray());
viewModel.Messages.Select(item => item.Message).OfType<WorkflowRulesEditorChatMessage>().ToArray());
pipeline.Release.SetResult();
await sendTask.WaitAsync(TimeSpan.FromSeconds(2));
Assert.False(viewModel.IsThinking);
Assert.Equal(2, viewModel.Messages.Count);
Assert.Equal(WorkflowRulesEditorChatRole.Agent, viewModel.Messages[1].Role);
Assert.Equal("Updated rules.", viewModel.Messages[1].Content);
AssertCompletedActivity(viewModel, ["Thinking..."], "Updated rules.");
}
[Fact]
@@ -881,20 +927,19 @@ public sealed class WorkflowRulesEditorTests
Assert.False(viewModel.IsThinking);
Assert.Equal("", viewModel.Draft);
Assert.Equal(
[
new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, "check logs"),
new WorkflowRulesEditorChatMessage(
WorkflowRulesEditorChatRole.Agent,
"Settings and logs failed: The request timed out.")
],
viewModel.Messages.ToArray());
Assert.Equal(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, "check logs"), viewModel.Messages[0].Message);
AssertCompletedActivity(
viewModel,
["Thinking..."],
"Meeting Summary Agent failed: The request timed out.");
}
[Fact]
public async Task ViewModelShowsReconnectStatusReportedByPipeline()
{
var pipeline = new StatusReportingRulesEditorPipeline("Updated rules.");
var pipeline = new ReportingRulesEditorPipeline(
"Updated rules.",
WorkflowRulesEditorActivityUpdate.Status("Reconnecting..."));
var viewModel = new WorkflowRulesEditorChatViewModel(pipeline)
{
Draft = "check logs"
@@ -911,7 +956,146 @@ public sealed class WorkflowRulesEditorTests
Assert.False(viewModel.IsThinking);
Assert.Equal("Thinking...", viewModel.ActivityMessage);
Assert.Equal("Updated rules.", viewModel.Messages[1].Content);
AssertCompletedActivity(viewModel, ["Thinking...", "Reconnecting..."], "Updated rules.");
}
[Fact]
public async Task ViewModelShowsToolCallsAbovePersistentThinkingLine()
{
var pipeline = new ReportingRulesEditorPipeline(
"Updated rules.",
WorkflowRulesEditorActivityUpdate.ToolCall("read_logs"));
var viewModel = new WorkflowRulesEditorChatViewModel(pipeline)
{
Draft = "check logs"
};
var sendTask = viewModel.SendAsync();
await pipeline.Reported.Task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.True(viewModel.IsThinking);
Assert.Equal("Thinking...", viewModel.ActivityMessage);
Assert.Equal(["Called tool: read_logs"], viewModel.ActivityMessages.ToArray());
pipeline.Release.SetResult();
await sendTask.WaitAsync(TimeSpan.FromSeconds(2));
Assert.False(viewModel.IsThinking);
Assert.Empty(viewModel.ActivityMessages);
AssertCompletedActivity(viewModel, ["Thinking...", "Called tool: read_logs"], "Updated rules.");
}
[Fact]
public async Task ViewModelShowsVisibleThinkingOutputInActivityExpander()
{
var pipeline = new ReportingRulesEditorPipeline(
"Updated rules.",
WorkflowRulesEditorActivityUpdate.Thinking("Checked the recent logs."),
WorkflowRulesEditorActivityUpdate.ToolCall("read_logs"),
WorkflowRulesEditorActivityUpdate.Thinking("Matched the rule to the user request."));
var viewModel = new WorkflowRulesEditorChatViewModel(pipeline)
{
Draft = "check logs"
};
var sendTask = viewModel.SendAsync();
await pipeline.Reported.Task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.True(viewModel.IsThinking);
Assert.Equal("Thinking...", viewModel.ActivityMessage);
Assert.Equal(
[
"Checked the recent logs.",
"Called tool: read_logs",
"Matched the rule to the user request."
],
viewModel.ActivityMessages.ToArray());
pipeline.Release.SetResult();
await sendTask.WaitAsync(TimeSpan.FromSeconds(2));
Assert.False(viewModel.IsThinking);
Assert.Empty(viewModel.ActivityMessages);
AssertCompletedActivity(
viewModel,
[
"Checked the recent logs.",
"Called tool: read_logs",
"Matched the rule to the user request."
],
"Updated rules.");
}
[Fact]
public async Task ViewModelMarshalsBackgroundActivityUpdatesToCapturedContext()
{
var previousContext = SynchronizationContext.Current;
var context = new RecordingSynchronizationContext();
var pipeline = new BackgroundReportingRulesEditorPipeline(
"Updated rules.",
WorkflowRulesEditorActivityUpdate.Thinking("Checked the recent logs."));
var viewModel = new WorkflowRulesEditorChatViewModel(pipeline)
{
Draft = "check logs"
};
SynchronizationContext.SetSynchronizationContext(context);
var sendTask = viewModel.SendAsync();
SynchronizationContext.SetSynchronizationContext(previousContext);
await pipeline.Reported.Task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.True(viewModel.IsThinking);
Assert.Equal(["Checked the recent logs."], viewModel.ActivityMessages.ToArray());
Assert.Equal(1, context.SendCount);
pipeline.Release.SetResult();
await sendTask.WaitAsync(TimeSpan.FromSeconds(2));
AssertCompletedActivity(viewModel, ["Checked the recent logs."], "Updated rules.");
}
[Fact]
public async Task ViewModelKeepsActivityItemsOutOfModelConversation()
{
var pipeline = new BlockingRulesEditorPipeline("Updated rules.");
var viewModel = new WorkflowRulesEditorChatViewModel(pipeline)
{
Draft = "first"
};
var firstSend = viewModel.SendAsync();
await pipeline.Started.Task.WaitAsync(TimeSpan.FromSeconds(2));
pipeline.Release.SetResult();
await firstSend.WaitAsync(TimeSpan.FromSeconds(2));
pipeline.Reset("Second response.");
viewModel.Draft = "second";
var secondSend = viewModel.SendAsync();
await pipeline.Started.Task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.Equal(
[
new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, "first"),
new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.Agent, "Updated rules.")
],
pipeline.LastConversation);
pipeline.Release.SetResult();
await secondSend.WaitAsync(TimeSpan.FromSeconds(2));
}
private static void AssertCompletedActivity(
WorkflowRulesEditorChatViewModel viewModel,
IReadOnlyList<string> expectedActivityLines,
string expectedAgentMessage)
{
Assert.Equal(3, viewModel.Messages.Count);
Assert.Equal(WorkflowRulesEditorConversationItemKind.Activity, viewModel.Messages[1].Kind);
Assert.StartsWith("Worked for ", viewModel.Messages[1].Content);
Assert.Equal(expectedActivityLines, viewModel.Messages[1].ActivityLines);
Assert.Equal(WorkflowRulesEditorConversationItemKind.Message, viewModel.Messages[2].Kind);
Assert.Equal(WorkflowRulesEditorChatRole.Agent, viewModel.Messages[2].Message?.Role);
Assert.Equal(expectedAgentMessage, viewModel.Messages[2].Message?.Content);
}
[Theory]
@@ -1196,31 +1380,36 @@ public sealed class WorkflowRulesEditorTests
private sealed class BlockingRulesEditorPipeline : IWorkflowRulesEditorChatPipeline
{
private readonly string response;
private string response;
public BlockingRulesEditorPipeline(string response)
{
this.response = response;
}
public TaskCompletionSource Started { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource Started { get; private set; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource Release { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource Release { get; private set; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public IReadOnlyList<WorkflowRulesEditorChatMessage> LastConversation { get; private set; } = [];
public void Reset(string nextResponse)
{
response = nextResponse;
Started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
Release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
}
public async Task<WorkflowRulesEditorChatResult> SendAsync(
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<string>? statusChanged = null)
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null)
{
LastConversation = conversation;
Started.SetResult();
await Release.Task.WaitAsync(cancellationToken);
return new WorkflowRulesEditorChatResult(
response,
conversation
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, userMessage))
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.Agent, response))
.ToList());
return new WorkflowRulesEditorChatResult(response);
}
}
@@ -1237,19 +1426,23 @@ public sealed class WorkflowRulesEditorTests
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<string>? statusChanged = null)
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null)
{
throw exception;
}
}
private sealed class StatusReportingRulesEditorPipeline : IWorkflowRulesEditorChatPipeline
private sealed class ReportingRulesEditorPipeline : IWorkflowRulesEditorChatPipeline
{
private readonly string response;
private readonly IReadOnlyList<WorkflowRulesEditorActivityUpdate> updates;
public StatusReportingRulesEditorPipeline(string response)
public ReportingRulesEditorPipeline(
string response,
params WorkflowRulesEditorActivityUpdate[] updates)
{
this.response = response;
this.updates = updates;
}
public TaskCompletionSource Reported { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
@@ -1260,17 +1453,57 @@ public sealed class WorkflowRulesEditorTests
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<string>? statusChanged = null)
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null)
{
statusChanged?.Invoke("Reconnecting...");
foreach (var update in updates)
{
activityChanged?.Invoke(update);
}
Reported.SetResult();
await Release.Task.WaitAsync(cancellationToken);
return new WorkflowRulesEditorChatResult(
response,
conversation
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, userMessage))
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.Agent, response))
.ToList());
return new WorkflowRulesEditorChatResult(response);
}
}
private sealed class BackgroundReportingRulesEditorPipeline : IWorkflowRulesEditorChatPipeline
{
private readonly string response;
private readonly WorkflowRulesEditorActivityUpdate update;
public BackgroundReportingRulesEditorPipeline(
string response,
WorkflowRulesEditorActivityUpdate update)
{
this.response = response;
this.update = update;
}
public TaskCompletionSource Reported { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource Release { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public async Task<WorkflowRulesEditorChatResult> SendAsync(
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null)
{
await Task.Run(() => activityChanged?.Invoke(update), cancellationToken);
Reported.SetResult();
await Release.Task.WaitAsync(cancellationToken);
return new WorkflowRulesEditorChatResult(response);
}
}
private sealed class RecordingSynchronizationContext : SynchronizationContext
{
public int SendCount { get; private set; }
public override void Send(SendOrPostCallback callback, object? state)
{
SendCount++;
callback(state);
}
}
@@ -98,7 +98,9 @@ public sealed class CalendarRecordingPromptScheduler : BackgroundService
meeting.Subject,
meeting.Start);
await promptService.ShowPromptAsync(
new MeetingStartPromptRequest(meeting),
new MeetingStartPromptRequest(
meeting,
recordingController.CurrentStatus.IsRecording && meeting.Metadata is not null),
(response, token) => HandlePromptResponseAsync(meeting, response, token),
cancellationToken);
}
@@ -139,6 +141,18 @@ public sealed class CalendarRecordingPromptScheduler : BackgroundService
MeetingStartPromptResponse response,
CancellationToken cancellationToken)
{
if (response == MeetingStartPromptResponse.AttachMetadataToCurrentMeeting)
{
if (meeting.Metadata is not null)
{
await recordingController.AttachMetadataToCurrentMeetingAsync(
meeting.Metadata,
cancellationToken);
}
return;
}
if (response != MeetingStartPromptResponse.Record)
{
return;
@@ -244,12 +258,15 @@ public interface IMeetingStartPromptService
CancellationToken cancellationToken);
}
public sealed record MeetingStartPromptRequest(CalendarMeeting Meeting);
public sealed record MeetingStartPromptRequest(
CalendarMeeting Meeting,
bool CanAttachToCurrentMeeting = false);
public enum MeetingStartPromptResponse
{
Record,
Skip
Skip,
AttachMetadataToCurrentMeeting
}
public interface IMeetingPromptRecordingController
@@ -262,6 +279,10 @@ public interface IMeetingPromptRecordingController
MeetingMetadata? metadata,
CancellationToken cancellationToken);
Task<RecordingStatus> AttachMetadataToCurrentMeetingAsync(
MeetingMetadata metadata,
CancellationToken cancellationToken);
Task<RecordingStatus> StopAsync(CancellationToken cancellationToken);
}
@@ -288,6 +309,13 @@ public sealed class MeetingPromptRecordingController : IMeetingPromptRecordingCo
return coordinator.StartFromPromptAsync(metadata, cancellationToken);
}
public Task<RecordingStatus> AttachMetadataToCurrentMeetingAsync(
MeetingMetadata metadata,
CancellationToken cancellationToken)
{
return coordinator.AttachMetadataToCurrentMeetingAsync(metadata, cancellationToken);
}
public Task<RecordingStatus> StopAsync(CancellationToken cancellationToken)
{
return coordinator.StopAsync(cancellationToken);
@@ -118,21 +118,10 @@ public sealed class WindowsMeetingStartPromptService : IMeetingStartPromptServic
string promptId,
MeetingStartPromptRequest request)
{
var yesButton = new ToastButton()
.SetContent("Yes")
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", "record")
.SetBackgroundActivation();
var yesButton = BuildResponseButton("Yes", promptId, "record");
var noButton = BuildResponseButton("No", promptId, "skip");
var noButton = new ToastButton()
.SetContent("No")
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", "skip")
.SetBackgroundActivation();
return new ToastContentBuilder()
var notification = new ToastContentBuilder()
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.SetToastScenario(ToastScenario.Reminder)
@@ -141,6 +130,30 @@ public sealed class WindowsMeetingStartPromptService : IMeetingStartPromptServic
.AddText($"{request.Meeting.Subject} starts at {request.Meeting.Start.LocalDateTime:t}.")
.AddButton(yesButton)
.AddButton(noButton);
if (request.CanAttachToCurrentMeeting)
{
notification.AddButton(
BuildResponseButton(
"Add metadata to current meeting",
promptId,
"attach-metadata"));
}
return notification;
}
private static ToastButton BuildResponseButton(
string content,
string promptId,
string response)
{
return new ToastButton()
.SetContent(content)
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", response)
.SetBackgroundActivation();
}
private void OnNotificationInvoked(ToastNotificationActivatedEventArgsCompat args)
@@ -166,10 +179,14 @@ public sealed class WindowsMeetingStartPromptService : IMeetingStartPromptServic
return;
}
var response = TryGetArgument(arguments, "response", out var responseValue) &&
string.Equals(responseValue, "record", StringComparison.OrdinalIgnoreCase)
? MeetingStartPromptResponse.Record
: MeetingStartPromptResponse.Skip;
var response = TryGetArgument(arguments, "response", out var responseValue)
? responseValue.ToLowerInvariant() switch
{
"record" => MeetingStartPromptResponse.Record,
"attach-metadata" => MeetingStartPromptResponse.AttachMetadataToCurrentMeeting,
_ => MeetingStartPromptResponse.Skip
}
: MeetingStartPromptResponse.Skip;
_ = Task.Run(async () =>
{
try
+7 -6
View File
@@ -20,19 +20,19 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.11.1" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.13.0" />
<PackageReference Include="DiffPlex" Version="1.9.0" />
<PackageReference Include="Microsoft.CognitiveServices.Speech" Version="$(MicrosoftSpeechVersion)" />
<PackageReference Include="Microsoft.CognitiveServices.Speech.Extension.MAS" Version="$(MicrosoftSpeechVersion)" ExcludeAssets="build" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.9" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.10" />
<PackageReference Include="NAudio" Version="2.3.0" />
<PackageReference Include="NCalcSync" Version="6.3.2" />
<PackageReference Include="NCalcSync" Version="6.4.0" />
<PackageReference Include="RazorLight" Version="2.3.1" />
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3" Version="3.0.3" />
<PackageReference Include="System.Drawing.Common" Version="10.0.9" />
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3" Version="3.0.5" />
<PackageReference Include="System.Drawing.Common" Version="10.0.10" />
<PackageReference Include="Whisper.net" Version="1.9.1" />
<PackageReference Include="Whisper.net.Runtime" Version="1.9.1" />
<PackageReference Include="YamlDotNet" Version="18.0.0" />
<PackageReference Include="YamlDotNet" Version="18.1.0" />
</ItemGroup>
<ItemGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'windows'">
@@ -65,6 +65,7 @@
<ItemGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) != 'windows'">
<Compile Remove="Hotkeys\GlobalHotkeyService.cs" />
<Compile Remove="Recording\NaudioCaptureSource.cs" />
<Compile Remove="Recording\WindowsMicrophoneDeviceProvider.cs" />
<Compile Remove="MeetingNotes\OutlookClassicMeetingMetadataProvider.Windows.cs" />
<Compile Remove="Screenshots\ActiveWindowScreenshotCapture.Windows.cs" />
<Compile Remove="Taskbar\UnoTaskbarIconService.Windows.cs" />
@@ -383,6 +383,8 @@ public sealed class AgentOptions
public string Model { get; set; } = "chatgpt/gpt-5.5";
public bool UseStreaming { get; set; } = true;
public bool EnableThinking { get; set; } = true;
public ReasoningEffortOption ReasoningEffort { get; set; } = ReasoningEffortOption.Medium;
@@ -414,6 +416,8 @@ public sealed class WorkflowRulesEditorOptions
public string? Model { get; set; }
public bool? UseStreaming { get; set; }
public bool? EnableThinking { get; set; }
public ReasoningEffortOption? ReasoningEffort { get; set; }
@@ -442,6 +446,7 @@ public sealed class WorkflowRulesEditorOptions
Key = string.IsNullOrWhiteSpace(Key) ? defaults.Key : Key,
KeyEnv = string.IsNullOrWhiteSpace(KeyEnv) ? defaults.KeyEnv : KeyEnv!,
Model = string.IsNullOrWhiteSpace(Model) ? defaults.Model : Model!,
UseStreaming = UseStreaming ?? defaults.UseStreaming,
EnableThinking = EnableThinking ?? defaults.EnableThinking,
ReasoningEffort = ReasoningEffort ?? defaults.ReasoningEffort,
ReconnectionAttempts = ReconnectionAttempts ?? defaults.ReconnectionAttempts,
@@ -1,4 +1,6 @@
using Microsoft.Extensions.Options;
using System.Globalization;
using System.Text;
using YamlDotNet.Core;
using YamlDotNet.Serialization;
@@ -113,7 +115,24 @@ public sealed class MarkdownMeetingNoteStore : IMeetingNoteStore
private static string EscapeQuoted(string value)
{
return $"\"{value.Replace("\"", "\\\"", StringComparison.Ordinal)}\"";
var escaped = new StringBuilder(value.Length + 2);
escaped.Append('"');
foreach (var character in value)
{
escaped.Append(character switch
{
'\\' => "\\\\",
'"' => "\\\"",
'\r' => "\\r",
'\n' => "\\n",
'\t' => "\\t",
< ' ' => "\\x" + ((int)character).ToString("X2", CultureInfo.InvariantCulture),
_ => character.ToString()
});
}
escaped.Append('"');
return escaped.ToString();
}
private static string EscapeNullableDateTime(DateTimeOffset? value)
@@ -135,7 +154,7 @@ public sealed class MarkdownMeetingNoteStore : IMeetingNoteStore
return "\"\"";
}
if (value.Contains(':', StringComparison.Ordinal) || value.StartsWith("[", StringComparison.Ordinal))
if (RequiresQuotedScalar(value))
{
return EscapeQuoted(value);
}
@@ -143,6 +162,49 @@ public sealed class MarkdownMeetingNoteStore : IMeetingNoteStore
return value;
}
private static bool RequiresQuotedScalar(string value)
{
if (char.IsWhiteSpace(value[0]) || char.IsWhiteSpace(value[^1]))
{
return true;
}
if (value.Contains(':', StringComparison.Ordinal) ||
value.Contains('\'', StringComparison.Ordinal) ||
value.Contains(" #", StringComparison.Ordinal) ||
value.Any(char.IsControl))
{
return true;
}
if (IsYamlIndicator(value[0]))
{
return true;
}
if (IsYamlCoreSchemaKeyword(value))
{
return true;
}
return DateTimeOffset.TryParse(value, out _) ||
double.TryParse(value, CultureInfo.InvariantCulture, out _);
}
private static bool IsYamlIndicator(char character)
{
return character is '-' or '?' or ':' or ',' or '[' or ']' or '{' or '}' or
'#' or '&' or '*' or '!' or '|' or '>' or '\'' or '"' or '%' or '@' or '`';
}
private static bool IsYamlCoreSchemaKeyword(string value)
{
return string.Equals(value, "true", StringComparison.OrdinalIgnoreCase) ||
string.Equals(value, "false", StringComparison.OrdinalIgnoreCase) ||
string.Equals(value, "null", StringComparison.OrdinalIgnoreCase) ||
value == "~";
}
private sealed class MeetingNoteYaml
{
[YamlMember(Alias = "title")]
+5 -1
View File
@@ -20,7 +20,11 @@ builder.Services.Configure<MeetingAssistantOptions>(builder.Configuration.GetSec
builder.Services.AddSingleton<ILaunchProfileOptionsProvider, ConfigurationLaunchProfileOptionsProvider>();
#if WINDOWS
builder.Services.AddSingleton<MicrophoneDeviceSelection>();
builder.Services.AddSingleton<IMicrophoneDeviceProvider, WindowsMicrophoneDeviceProvider>();
builder.Services.AddSingleton<WindowsMicrophoneDeviceProvider>();
builder.Services.AddSingleton<IMicrophoneDeviceProvider>(services =>
services.GetRequiredService<WindowsMicrophoneDeviceProvider>());
builder.Services.AddSingleton<IMicrophoneCaptureSourceFactory>(services =>
services.GetRequiredService<WindowsMicrophoneDeviceProvider>());
builder.Services.AddSingleton<MicrophoneAudioSource>();
builder.Services.AddSingleton<SystemAudioSource>();
builder.Services.AddSingleton<IAcousticEchoCancellerFactory, AdaptiveFilterAcousticEchoCancellerFactory>();
@@ -0,0 +1,6 @@
namespace MeetingAssistant.Recording;
public interface IMicrophoneCaptureSourceFactory
{
IMeetingAudioSource CreateCapture(MeetingAssistantOptions options);
}
@@ -1,5 +1,3 @@
using NAudio.Wave;
namespace MeetingAssistant.Recording;
public interface IMicrophoneDeviceProvider
@@ -7,6 +5,4 @@ public interface IMicrophoneDeviceProvider
IReadOnlyList<MicrophoneDevice> GetAvailableMicrophones();
MicrophoneDeviceSnapshot GetMicrophoneSnapshot(MeetingAssistantOptions options);
IWaveIn CreateCapture(MeetingAssistantOptions options);
}
@@ -155,6 +155,44 @@ public sealed class MeetingRecordingCoordinator
return await StartAsync(null, metadata, suppressMetadataLookup: true, cancellationToken);
}
public async Task<RecordingStatus> AttachMetadataToCurrentMeetingAsync(
MeetingMetadata metadata,
CancellationToken cancellationToken)
{
await gate.WaitAsync(cancellationToken);
try
{
var run = currentRun;
if (run is null || run.IsCaptureStopping)
{
return CurrentStatus;
}
var meetingNote = await ApplyAndPersistMeetingMetadataAsync(
run.Artifacts,
await meetingNoteStore.ReadAsync(run.MeetingNotePath, cancellationToken),
metadata,
run.Options,
cancellationToken);
currentMeetingNote = meetingNote;
await transcriptStore.UpdateMetadataAsync(
run.Session,
run.Artifacts,
meetingNote,
cancellationToken);
run.MarkPromptMetadataAttached();
logger.LogInformation(
"Attached prompted calendar metadata to active meeting {MeetingNotePath}",
run.MeetingNotePath);
return CurrentStatus;
}
finally
{
gate.Release();
}
}
public async Task<RecordingStatus> StartAsync(
string? launchProfileName,
CancellationToken cancellationToken)
@@ -213,16 +251,11 @@ public sealed class MeetingRecordingCoordinator
currentMeetingNote = await meetingNoteStore.ReadAsync(currentMeetingNote.Path, cancellationToken);
if (suppliedMetadata is not null)
{
await ApplyMeetingMetadataAsync(currentMeetingNote, suppliedMetadata, runOptions, cancellationToken);
currentMeetingNote = await meetingNoteStore.SaveAsync(
currentMeetingNote,
runOptions,
cancellationToken);
await meetingArtifactStore.UpdateAssistantContextMetadataAsync(
currentMeetingNote = await ApplyAndPersistMeetingMetadataAsync(
currentArtifacts,
currentMeetingNote,
suppliedMetadata.Agenda,
suppliedMetadata.ScheduledEnd,
suppliedMetadata,
runOptions,
cancellationToken);
}
@@ -1016,7 +1049,7 @@ public sealed class MeetingRecordingCoordinator
return;
}
if (run.IsAborted)
if (run.IsAborted || run.HasAttachedPromptMetadata)
{
return;
}
@@ -1024,25 +1057,21 @@ public sealed class MeetingRecordingCoordinator
await gate.WaitAsync(CancellationToken.None);
try
{
if (run.IsAborted)
if (run.IsAborted || run.HasAttachedPromptMetadata)
{
return;
}
var meetingNote = await meetingNoteStore.ReadAsync(run.MeetingNotePath, CancellationToken.None);
await ApplyMeetingMetadataAsync(meetingNote, metadata, run.Options, CancellationToken.None);
meetingNote = await meetingNoteStore.SaveAsync(meetingNote, run.Options, CancellationToken.None);
var meetingNote = await ApplyAndPersistMeetingMetadataAsync(
run.Artifacts,
await meetingNoteStore.ReadAsync(run.MeetingNotePath, CancellationToken.None),
metadata,
run.Options,
CancellationToken.None);
if (currentMeetingNote?.Path == meetingNote.Path)
{
currentMeetingNote = meetingNote;
}
await meetingArtifactStore.UpdateAssistantContextMetadataAsync(
run.Artifacts,
meetingNote,
metadata.Agenda,
metadata.ScheduledEnd,
CancellationToken.None);
logger.LogInformation(
"Applied Outlook meeting metadata to {MeetingNotePath}",
run.MeetingNotePath);
@@ -1082,6 +1111,7 @@ public sealed class MeetingRecordingCoordinator
}
private async Task ApplyMeetingMetadataAsync(
MeetingSessionArtifacts artifacts,
MeetingNote meetingNote,
MeetingMetadata metadata,
MeetingAssistantOptions options,
@@ -1104,10 +1134,65 @@ public sealed class MeetingRecordingCoordinator
return;
}
meetingNote.Frontmatter.Attendees = await CanonicalizeAttendeesAsync(metadata.Attendees, cancellationToken);
var canonicalized = await CanonicalizeAttendeesAsync(metadata.Attendees, cancellationToken);
meetingNote.Frontmatter.Attendees = await TransformAttendeesAsync(
artifacts,
canonicalized,
options,
cancellationToken);
}
}
private async Task<MeetingNote> ApplyAndPersistMeetingMetadataAsync(
MeetingSessionArtifacts artifacts,
MeetingNote meetingNote,
MeetingMetadata metadata,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
await ApplyMeetingMetadataAsync(
artifacts,
meetingNote,
metadata,
options,
cancellationToken);
var savedMeetingNote = await meetingNoteStore.SaveAsync(
meetingNote,
options,
cancellationToken);
await meetingArtifactStore.UpdateAssistantContextMetadataAsync(
artifacts,
savedMeetingNote,
metadata.Agenda,
metadata.ScheduledEnd,
cancellationToken);
return savedMeetingNote;
}
private async Task<List<string>> TransformAttendeesAsync(
MeetingSessionArtifacts artifacts,
IReadOnlyList<string> attendees,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
var transformed = new List<string>();
foreach (var attendee in attendees)
{
var value = await TransformAttendeeAsync(artifacts, attendee, options, cancellationToken);
var storageValue = string.Equals(value, attendee, StringComparison.Ordinal)
? attendee.Trim()
: NormalizeAttendeeName(value);
var normalized = NormalizeAttendeeName(storageValue);
if (!string.IsNullOrWhiteSpace(normalized) &&
!transformed.Contains(normalized, StringComparer.OrdinalIgnoreCase))
{
transformed.Add(storageValue);
}
}
return transformed;
}
private async Task<List<string>> CanonicalizeAttendeesAsync(
IReadOnlyList<string> attendees,
CancellationToken cancellationToken)
@@ -1422,8 +1507,17 @@ public sealed class MeetingRecordingCoordinator
continue;
}
var transformedDisplayName = await TransformAttendeeAsync(
artifacts,
match.DisplayName.Trim(),
runOptions,
cancellationToken);
var displayName = string.Equals(transformedDisplayName, match.DisplayName.Trim(), StringComparison.Ordinal)
? match.DisplayName.Trim()
: NormalizeAttendeeName(transformedDisplayName);
var acceptedNames = match.AcceptedNames
.Append(match.DisplayName)
.Append(displayName)
.Select(NormalizeAttendeeName)
.Where(name => !string.IsNullOrWhiteSpace(name))
.Distinct(StringComparer.OrdinalIgnoreCase)
@@ -1442,7 +1536,6 @@ public sealed class MeetingRecordingCoordinator
continue;
}
var displayName = match.DisplayName.Trim();
meetingNote.Frontmatter.Attendees.Add(displayName);
existingNames.Add(NormalizeAttendeeName(displayName));
changed = true;
@@ -1502,6 +1595,18 @@ public sealed class MeetingRecordingCoordinator
return MeetingAttendeeNames.NormalizeDisplayName(attendee);
}
private async Task<string> TransformAttendeeAsync(
MeetingSessionArtifacts artifacts,
string attendee,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return await meetingWorkflowEngine.TransformAttendeeAsync(
MeetingWorkflowEvent.AttendeeAdded(artifacts, attendee),
options,
cancellationToken);
}
private static bool RemoveDuplicateAcceptedAliases(
List<string> attendees,
string displayName,
@@ -1853,6 +1958,8 @@ public sealed class MeetingRecordingCoordinator
public bool HasSwitchedProfile { get; private set; }
public bool HasAttachedPromptMetadata { get; private set; }
public AssistantContextState ContextState { get; private set; } = AssistantContextState.CollectingMetadata;
public DateTimeOffset? InferredEndTime
@@ -1871,6 +1978,11 @@ public sealed class MeetingRecordingCoordinator
CaptureCancellationSource.Cancel();
}
public void MarkPromptMetadataAttached()
{
HasAttachedPromptMetadata = true;
}
public void Abort()
{
IsAborted = true;
@@ -0,0 +1,145 @@
using System.Runtime.CompilerServices;
namespace MeetingAssistant.Recording;
public sealed class MicrophoneAudioSource : IMeetingAudioSource
{
private static readonly TimeSpan DefaultRecoveryDelay = TimeSpan.FromSeconds(1);
private readonly IMicrophoneCaptureSourceFactory captureSources;
private readonly ILogger<MicrophoneAudioSource> logger;
private readonly TimeSpan recoveryDelay;
public MicrophoneAudioSource(
IMicrophoneCaptureSourceFactory captureSources,
ILogger<MicrophoneAudioSource> logger)
: this(captureSources, logger, DefaultRecoveryDelay)
{
}
internal MicrophoneAudioSource(
IMicrophoneCaptureSourceFactory captureSources,
ILogger<MicrophoneAudioSource> logger,
TimeSpan recoveryDelay)
{
this.captureSources = captureSources;
this.logger = logger;
this.recoveryDelay = recoveryDelay;
}
public IAsyncEnumerable<AudioChunk> CaptureAsync(CancellationToken cancellationToken)
{
return CaptureAsync(new MeetingAssistantOptions(), cancellationToken);
}
public async IAsyncEnumerable<AudioChunk> CaptureAsync(
MeetingAssistantOptions options,
[EnumeratorCancellation] CancellationToken cancellationToken)
{
var failedAttempts = 0;
while (!cancellationToken.IsCancellationRequested)
{
IAsyncEnumerator<AudioChunk>? capture = null;
Exception? failure = null;
try
{
capture = captureSources
.CreateCapture(options)
.CaptureAsync(options, cancellationToken)
.GetAsyncEnumerator(cancellationToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
catch (Exception exception)
{
failure = exception;
}
if (cancellationToken.IsCancellationRequested)
{
yield break;
}
if (capture is not null)
{
try
{
while (!cancellationToken.IsCancellationRequested)
{
var hasNext = false;
try
{
hasNext = await capture.MoveNextAsync();
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
catch (Exception exception)
{
failure = exception;
}
if (cancellationToken.IsCancellationRequested || failure is not null || !hasNext)
{
break;
}
if (failedAttempts > 0)
{
logger.LogInformation(
"Microphone capture recovered after {FailedAttemptCount} failed attempt(s)",
failedAttempts);
failedAttempts = 0;
}
yield return capture.Current;
}
}
finally
{
try
{
await capture.DisposeAsync();
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
catch (Exception exception)
{
failure ??= exception;
}
}
}
if (cancellationToken.IsCancellationRequested)
{
yield break;
}
failedAttempts++;
logger.LogWarning(
failure,
"Microphone capture stopped unexpectedly; re-resolving an available microphone in {RecoveryDelay}",
recoveryDelay);
if (!await WaitForRecoveryAsync(cancellationToken))
{
yield break;
}
}
}
private async Task<bool> WaitForRecoveryAsync(CancellationToken cancellationToken)
{
try
{
await Task.Delay(recoveryDelay, cancellationToken);
return true;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
return false;
}
}
}
@@ -34,6 +34,8 @@ public sealed class MicrophoneDeviceSelection
var selected = SelectedDeviceId;
return FindById(availableDevices, selected) ??
FindById(availableDevices, configuredDeviceId) ??
FindById(availableDevices, defaultDevice?.Id) ??
availableDevices.FirstOrDefault() ??
defaultDevice;
}
@@ -4,41 +4,28 @@ using NAudio.Wave;
namespace MeetingAssistant.Recording;
public sealed class MicrophoneAudioSource : IMeetingAudioSource
internal sealed class NaudioCaptureAudioSource : IMeetingAudioSource
{
private readonly IMicrophoneDeviceProvider microphones;
private readonly ILogger<MicrophoneAudioSource> logger;
private readonly IWaveIn capture;
private readonly string sourceName;
private readonly ILogger logger;
public MicrophoneAudioSource(
IMicrophoneDeviceProvider microphones,
ILogger<MicrophoneAudioSource> logger)
public NaudioCaptureAudioSource(
IWaveIn capture,
string sourceName,
ILogger logger)
{
this.microphones = microphones;
this.capture = capture;
this.sourceName = sourceName;
this.logger = logger;
}
public IAsyncEnumerable<AudioChunk> CaptureAsync(CancellationToken cancellationToken)
{
return CaptureAsync(new MeetingAssistantOptions(), cancellationToken);
return CaptureWith(capture, sourceName, logger, cancellationToken);
}
public IAsyncEnumerable<AudioChunk> CaptureAsync(
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return CaptureAsync(microphones.CreateCapture(options), options, cancellationToken);
}
private IAsyncEnumerable<AudioChunk> CaptureAsync(
IWaveIn capture,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
capture.WaveFormat = new WaveFormat(options.Recording.SampleRate, 16, options.Recording.Channels);
return CaptureWith(capture, "microphone", logger, cancellationToken);
}
internal static async IAsyncEnumerable<AudioChunk> CaptureWith(
private static async IAsyncEnumerable<AudioChunk> CaptureWith(
IWaveIn capture,
string sourceName,
ILogger logger,
@@ -124,6 +111,6 @@ public sealed class SystemAudioSource : IMeetingAudioSource
WaveFormat = new WaveFormat(options.Recording.SampleRate, 16, options.Recording.Channels)
};
return MicrophoneAudioSource.CaptureWith(capture, "system", logger, cancellationToken);
return new NaudioCaptureAudioSource(capture, "system", logger).CaptureAsync(cancellationToken);
}
}
@@ -3,7 +3,7 @@ using NAudio.Wave;
namespace MeetingAssistant.Recording;
public sealed class WindowsMicrophoneDeviceProvider : IMicrophoneDeviceProvider
public sealed class WindowsMicrophoneDeviceProvider : IMicrophoneDeviceProvider, IMicrophoneCaptureSourceFactory
{
private readonly MicrophoneDeviceSelection selection;
private readonly ILogger<WindowsMicrophoneDeviceProvider> logger;
@@ -42,21 +42,38 @@ public sealed class WindowsMicrophoneDeviceProvider : IMicrophoneDeviceProvider
selection.Resolve(options.Recording.MicrophoneDeviceId, GetDefaultMicrophone(), devices));
}
public IWaveIn CreateCapture(MeetingAssistantOptions options)
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options)
{
var current = GetMicrophoneSnapshot(options).Current;
IWaveIn capture;
if (current is null)
{
logger.LogInformation("Starting microphone capture from Windows default capture endpoint");
return new WasapiCapture();
capture = new WasapiCapture();
}
else
{
logger.LogInformation(
"Starting microphone capture from {MicrophoneName} ({MicrophoneDeviceId})",
current.Name,
current.Id);
using var enumerator = new MMDeviceEnumerator();
capture = new WasapiCapture(enumerator.GetDevice(current.Id));
}
logger.LogInformation(
"Starting microphone capture from {MicrophoneName} ({MicrophoneDeviceId})",
current.Name,
current.Id);
using var enumerator = new MMDeviceEnumerator();
return new WasapiCapture(enumerator.GetDevice(current.Id));
try
{
capture.WaveFormat = new WaveFormat(
options.Recording.SampleRate,
16,
options.Recording.Channels);
return new NaudioCaptureAudioSource(capture, "microphone", logger);
}
catch
{
capture.Dispose();
throw;
}
}
private static MicrophoneDevice? GetDefaultMicrophone()
@@ -1,15 +1,13 @@
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Summary;
using Microsoft.Extensions.AI;
namespace MeetingAssistant.Screenshots;
public sealed partial class LiteLlmScreenshotOcrClient : IScreenshotOcrClient
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
private readonly ILogger<LiteLlmScreenshotOcrClient> logger;
private readonly Func<HttpMessageHandler>? httpMessageHandlerFactory;
@@ -40,20 +38,30 @@ public sealed partial class LiteLlmScreenshotOcrClient : IScreenshotOcrClient
: options.Agent.Model;
var key = ResolveApiKey(options);
var imageBytes = await File.ReadAllBytesAsync(screenshotPath, cancellationToken);
using var httpClient = CreateHttpClient();
httpClient.BaseAddress = NormalizeEndpoint(new Uri(endpoint));
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", key);
var payload = CreatePayload(model, CreatePrompt(prompt, imageBytes), imageBytes);
using var content = new StringContent(payload.ToJsonString(JsonOptions), Encoding.UTF8, "application/json");
using var response = await httpClient.PostAsync("responses", content, cancellationToken);
var responseJson = await response.Content.ReadAsStringAsync(cancellationToken);
if (!response.IsSuccessStatusCode)
{
throw new InvalidOperationException(
$"Screenshot OCR request failed with {(int)response.StatusCode} {response.ReasonPhrase}: {responseJson}");
}
var text = ParseOutputText(responseJson);
var httpClient = CreateHttpClient();
httpClient.BaseAddress = LiteLlmResponsesChatClient.NormalizeEndpoint(new Uri(endpoint));
using var chatClient = new LiteLlmResponsesChatClient(
httpClient,
key,
model,
enableThinking: false,
reasoningEffort: "none",
reconnectionAttempts: options.Agent.ReconnectionAttempts,
reconnectionDelay: options.Agent.ReconnectionDelay,
logger: logger,
firstRequestIsUser: false,
useStreaming: options.Agent.UseStreaming);
var response = await chatClient.GetResponseAsync(
[
new ChatMessage(
ChatRole.User,
[
new TextContent(CreatePrompt(prompt, imageBytes)),
new DataContent(imageBytes, "image/png")
])
],
cancellationToken: cancellationToken);
var text = response.Text ?? string.Empty;
logger.LogInformation("Screenshot OCR completed for {ScreenshotPath}", screenshotPath);
return ParseOcrResult(text);
}
@@ -65,35 +73,6 @@ public sealed partial class LiteLlmScreenshotOcrClient : IScreenshotOcrClient
: new HttpClient(httpMessageHandlerFactory());
}
private static JsonObject CreatePayload(string model, string prompt, byte[] imageBytes)
{
return new JsonObject
{
["model"] = model,
["store"] = false,
["input"] = new JsonArray
{
new JsonObject
{
["role"] = "user",
["content"] = new JsonArray
{
new JsonObject
{
["type"] = "input_text",
["text"] = prompt
},
new JsonObject
{
["type"] = "input_image",
["image_url"] = "data:image/png;base64," + Convert.ToBase64String(imageBytes)
}
}
}
}
};
}
private static string CreatePrompt(string prompt, byte[] imageBytes)
{
return TryReadPngDimensions(imageBytes, out var width, out var height)
@@ -210,46 +189,6 @@ public sealed partial class LiteLlmScreenshotOcrClient : IScreenshotOcrClient
bytes[offset + 3];
}
private static string ParseOutputText(string responseJson)
{
using var document = JsonDocument.Parse(responseJson);
var root = document.RootElement;
var parts = new List<string>();
if (root.TryGetProperty("output_text", out var outputText) &&
outputText.ValueKind == JsonValueKind.String &&
!string.IsNullOrWhiteSpace(outputText.GetString()))
{
parts.Add(outputText.GetString()!);
}
if (root.TryGetProperty("output", out var output) && output.ValueKind == JsonValueKind.Array)
{
foreach (var item in output.EnumerateArray())
{
if (!item.TryGetProperty("content", out var content) || content.ValueKind != JsonValueKind.Array)
{
continue;
}
foreach (var block in content.EnumerateArray())
{
if (block.TryGetProperty("type", out var type) &&
type.GetString() == "output_text" &&
block.TryGetProperty("text", out var text) &&
text.ValueKind == JsonValueKind.String &&
!string.IsNullOrWhiteSpace(text.GetString()))
{
parts.Add(text.GetString()!);
}
}
}
}
return parts.Count == 0
? ""
: string.Join(Environment.NewLine + Environment.NewLine, parts);
}
private static string ResolveApiKey(MeetingAssistantOptions options)
{
if (!string.IsNullOrWhiteSpace(options.Screenshots.Ocr.Key))
@@ -278,17 +217,6 @@ public sealed partial class LiteLlmScreenshotOcrClient : IScreenshotOcrClient
$"No screenshot OCR API key configured. Set MeetingAssistant:Screenshots:Ocr:Key or environment variable '{options.Screenshots.Ocr.KeyEnv}'.");
}
private static Uri NormalizeEndpoint(Uri endpoint)
{
var value = endpoint.ToString().TrimEnd('/');
if (!value.EndsWith("/v1", StringComparison.OrdinalIgnoreCase))
{
value += "/v1";
}
return new Uri(value + "/");
}
[GeneratedRegex("```json\\s*(?<json>.*?)\\s*```", RegexOptions.Singleline | RegexOptions.IgnoreCase)]
private static partial Regex JsonCodeBlockRegex();
}
@@ -5,6 +5,7 @@ using System.Runtime.InteropServices;
using System.Text.RegularExpressions;
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Speakers;
using MeetingAssistant.Workflow;
namespace MeetingAssistant.Screenshots;
@@ -149,6 +150,7 @@ public sealed partial class MeetingScreenshotService :
private readonly IMeetingArtifactStore artifactStore;
private readonly IMeetingNoteStore meetingNoteStore;
private readonly ISpeakerIdentityAttendeeCanonicalizer attendeeCanonicalizer;
private readonly IMeetingWorkflowEngine meetingWorkflowEngine;
private readonly IScreenshotOcrClient ocrClient;
private readonly ILogger<MeetingScreenshotService> logger;
private readonly ConcurrentDictionary<string, List<PendingOcrTask>> pendingOcrByContext = new(StringComparer.OrdinalIgnoreCase);
@@ -160,12 +162,14 @@ public sealed partial class MeetingScreenshotService :
IMeetingNoteStore meetingNoteStore,
ISpeakerIdentityAttendeeCanonicalizer attendeeCanonicalizer,
IScreenshotOcrClient ocrClient,
ILogger<MeetingScreenshotService> logger)
ILogger<MeetingScreenshotService> logger,
IMeetingWorkflowEngine? meetingWorkflowEngine = null)
{
this.screenshotCapture = screenshotCapture;
this.artifactStore = artifactStore;
this.meetingNoteStore = meetingNoteStore;
this.attendeeCanonicalizer = attendeeCanonicalizer;
this.meetingWorkflowEngine = meetingWorkflowEngine ?? NoopMeetingWorkflowEngine.Instance;
this.ocrClient = ocrClient;
this.logger = logger;
}
@@ -434,6 +438,7 @@ public sealed partial class MeetingScreenshotService :
await AddOcrAttendeesAsync(
artifacts,
result.Attendees,
options,
CancellationToken.None);
}
}
@@ -477,6 +482,7 @@ public sealed partial class MeetingScreenshotService :
private async Task AddOcrAttendeesAsync(
MeetingSessionArtifacts artifacts,
IReadOnlyList<string> attendees,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
var additions = MeetingAttendeeNames.NormalizeDistinct(attendees);
@@ -488,8 +494,13 @@ public sealed partial class MeetingScreenshotService :
try
{
var meetingNote = await meetingNoteStore.ReadAsync(artifacts.MeetingNotePath, cancellationToken);
var transformedAdditions = await TransformAttendeesAsync(
artifacts,
additions,
options,
cancellationToken);
var canonicalized = await attendeeCanonicalizer.CanonicalizeAsync(
meetingNote.Frontmatter.Attendees.Concat(additions).ToList(),
meetingNote.Frontmatter.Attendees.Concat(transformedAdditions).ToList(),
cancellationToken);
if (meetingNote.Frontmatter.Attendees.SequenceEqual(canonicalized, StringComparer.Ordinal))
{
@@ -512,6 +523,30 @@ public sealed partial class MeetingScreenshotService :
}
}
private async Task<IReadOnlyList<string>> TransformAttendeesAsync(
MeetingSessionArtifacts artifacts,
IReadOnlyList<string> attendees,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
var transformed = new List<string>();
foreach (var attendee in attendees)
{
var value = await meetingWorkflowEngine.TransformAttendeeAsync(
MeetingWorkflowEvent.AttendeeAdded(artifacts, attendee),
options,
cancellationToken);
var normalized = MeetingAttendeeNames.NormalizeDisplayName(value);
if (!string.IsNullOrWhiteSpace(normalized) &&
!transformed.Contains(normalized, StringComparer.OrdinalIgnoreCase))
{
transformed.Add(normalized);
}
}
return transformed;
}
private async Task<bool> ReplaceOcrPlaceholderAsync(
string assistantContextPath,
string screenshotId,
@@ -0,0 +1,26 @@
using System.Text.Json.Nodes;
using Microsoft.Extensions.AI;
namespace MeetingAssistant.Summary;
internal static class FunctionInvocationGuard
{
public static ValueTask<object?> InvokeAsync(
FunctionInvocationContext context,
CancellationToken cancellationToken)
{
if (context.CallContent.Exception is not null)
{
return ValueTask.FromResult<object?>(new JsonObject
{
["error"] = new JsonObject
{
["code"] = "invalid_tool_arguments",
["message"] = "Tool arguments must be a valid JSON object."
}
});
}
return context.Function.InvokeAsync(context.Arguments, cancellationToken);
}
}
@@ -1,20 +1,28 @@
using System.Net.Http.Headers;
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
using OpenAI;
using OpenAI.Responses;
namespace MeetingAssistant.Summary;
#pragma warning disable MAAI001
#pragma warning disable OPENAI001
public sealed class LiteLlmResponsesChatClient : IChatClient
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
private readonly HttpClient httpClient;
private readonly ResponsesClient responsesClient;
private readonly AsyncLocal<string?> requestInitiator = new();
private readonly string model;
private readonly bool useStreaming;
private readonly bool enableThinking;
private readonly string reasoningEffort;
private readonly int reconnectionAttempts;
@@ -22,6 +30,7 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
private readonly LiteLlmResponsesCompactionOptions? compactionOptions;
private readonly ILogger? logger;
private readonly Action? retrying;
private readonly Action<string>? reasoningSummaryChanged;
private int responsesRequestCount;
public LiteLlmResponsesChatClient(
@@ -35,7 +44,9 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
LiteLlmResponsesCompactionOptions? compactionOptions = null,
ILogger? logger = null,
bool firstRequestIsUser = true,
Action? retrying = null)
Action? retrying = null,
Action<string>? reasoningSummaryChanged = null,
bool useStreaming = true)
: this(
new HttpClient { BaseAddress = NormalizeEndpoint(endpoint) },
apiKey,
@@ -47,7 +58,9 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
compactionOptions,
logger,
firstRequestIsUser,
retrying)
retrying,
reasoningSummaryChanged,
useStreaming)
{
}
@@ -62,11 +75,15 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
LiteLlmResponsesCompactionOptions? compactionOptions = null,
ILogger? logger = null,
bool firstRequestIsUser = true,
Action? retrying = null)
Action? retrying = null,
Action<string>? reasoningSummaryChanged = null,
bool useStreaming = true)
{
this.httpClient = httpClient;
this.httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
responsesClient = CreateResponsesClient(httpClient, apiKey, requestInitiator);
this.model = model;
this.useStreaming = useStreaming;
this.enableThinking = enableThinking;
this.reasoningEffort = reasoningEffort;
this.reconnectionAttempts = Math.Max(0, reconnectionAttempts);
@@ -74,6 +91,7 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
this.compactionOptions = compactionOptions;
this.logger = logger;
this.retrying = retrying;
this.reasoningSummaryChanged = reasoningSummaryChanged;
responsesRequestCount = firstRequestIsUser ? 0 : 1;
}
@@ -82,19 +100,50 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
httpClient.Dispose();
}
internal static LiteLlmResponsesChatClient Create(
AgentOptions options,
string apiKey,
LiteLlmResponsesCompactionOptions? compactionOptions,
ILogger? logger,
bool firstRequestIsUser,
Action? retrying = null,
Action<string>? reasoningSummaryChanged = null)
{
return new LiteLlmResponsesChatClient(
new Uri(options.Endpoint),
apiKey,
options.Model,
options.EnableThinking,
options.ReasoningEffort switch
{
ReasoningEffortOption.None => "none",
ReasoningEffortOption.Low => "low",
ReasoningEffortOption.High => "high",
ReasoningEffortOption.ExtraHigh => "xhigh",
_ => "medium"
},
options.ReconnectionAttempts,
options.ReconnectionDelay,
compactionOptions,
logger,
firstRequestIsUser,
retrying,
reasoningSummaryChanged,
options.UseStreaming);
}
public async Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken cancellationToken = default)
{
var payload = await CreateCompactedPayloadAsync(messages.ToList(), options, cancellationToken).ConfigureAwait(false);
var responseJson = await PostWithRetryAsync(payload, cancellationToken).ConfigureAwait(false);
var response = ParseResponseJson(responseJson);
LogResponseDiagnostics(responseJson, response);
ThrowIfResponseHasNoContent(responseJson, response);
LogResponseUsage(response.Usage);
return response;
var result = await PostWithRetryAsync(payload, cancellationToken).ConfigureAwait(false);
ReportVisibleReasoningSummaries(result.ReasoningSummaries);
LogResponseDiagnostics(result.Response);
ThrowIfResponseHasNoContent(result.Response);
LogResponseUsage(result.Response.Usage);
return result.Response;
}
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
@@ -121,45 +170,36 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
: null;
}
public static ChatResponse ParseResponseJson(string responseJson)
private static IReadOnlyList<string> ExtractVisibleReasoningSummaries(ChatResponse response)
{
using var document = JsonDocument.Parse(responseJson);
var root = document.RootElement;
var contents = new List<AIContent>();
return response.Messages
.SelectMany(message => message.Contents)
.OfType<TextReasoningContent>()
.Select(content => content.Text)
.Where(text => !string.IsNullOrWhiteSpace(text))
.ToArray();
}
if (root.TryGetProperty("output", out var output) && output.ValueKind == JsonValueKind.Array)
private void ReportVisibleReasoningSummaries(IReadOnlyList<string> summaries)
{
if (reasoningSummaryChanged is null)
{
foreach (var item in output.EnumerateArray())
{
var type = GetString(item, "type");
if (type == "message")
{
AddMessageContent(contents, item);
}
else if (type == "function_call")
{
contents.Add(new FunctionCallContent(
GetRequiredString(item, "call_id"),
GetRequiredString(item, "name"),
ParseArguments(GetString(item, "arguments"))));
}
}
return;
}
var message = new ChatMessage
foreach (var summary in summaries)
{
Role = ChatRole.Assistant,
Contents = contents
};
return new ChatResponse(message)
{
ResponseId = GetString(root, "id"),
ModelId = GetString(root, "model"),
CreatedAt = GetUnixTimestamp(root, "created_at"),
Usage = ParseUsage(root),
RawRepresentation = responseJson
};
try
{
reasoningSummaryChanged(summary);
}
catch (Exception exception)
{
logger?.LogWarning(
exception,
"Reasoning summary callback failed; continuing with the LiteLLM response.");
}
}
}
private async Task<JsonObject> CreateCompactedPayloadAsync(
@@ -345,7 +385,9 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
return payload;
}
private async Task<string> PostWithRetryAsync(JsonObject payload, CancellationToken cancellationToken)
private async Task<ResponsesChatResult> PostWithRetryAsync(
JsonObject payload,
CancellationToken cancellationToken)
{
var payloadJson = payload.ToJsonString(JsonOptions);
Exception? lastException = null;
@@ -356,26 +398,36 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
{
var initiator = NextInitiator();
LogRequestDiagnostics(payload, initiator, attempt);
using var request = CreateJsonRequest(
"responses",
payloadJson,
initiator);
using var response = await httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false);
var responseJson = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
var createOptions = ModelReaderWriter.Read<CreateResponseOptions>(
BinaryData.FromString(payloadJson),
ModelReaderWriterOptions.Json)
?? throw new InvalidOperationException("Unable to create OpenAI Responses request options.");
createOptions.StreamingEnabled = useStreaming;
var previousInitiator = requestInitiator.Value;
requestInitiator.Value = initiator;
try
{
return responseJson;
return useStreaming
? await GetStreamingResponseAsync(createOptions, cancellationToken).ConfigureAwait(false)
: CreateNonStreamingResult((await responsesClient
.CreateResponseAsync(createOptions, cancellationToken)
.ConfigureAwait(false))
.Value
.AsChatResponse(createOptions));
}
var exception = new InvalidOperationException(
$"LiteLLM Responses request failed with {(int)response.StatusCode} {response.ReasonPhrase}: {responseJson}");
if (!IsRetryableStatusCode((int)response.StatusCode) || attempt == reconnectionAttempts)
finally
{
throw exception;
requestInitiator.Value = previousInitiator;
}
lastException = exception;
}
catch (ClientResultException exception)
when (!IsRetryableStatusCode(exception.Status) || attempt == reconnectionAttempts)
{
throw new InvalidOperationException(
$"LiteLLM Responses request failed with {exception.Status}: {exception.Message}",
exception);
}
catch (Exception exception) when (IsRetryableException(exception) && attempt < reconnectionAttempts)
{
@@ -389,6 +441,48 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
throw lastException ?? new InvalidOperationException("LiteLLM Responses request failed.");
}
private async Task<ResponsesChatResult> GetStreamingResponseAsync(
CreateResponseOptions createOptions,
CancellationToken cancellationToken)
{
var updates = new List<ChatResponseUpdate>();
var reasoningSummaries = new Dictionary<(string ItemId, int SummaryIndex), StringBuilder>();
await foreach (var update in responsesClient
.CreateResponseStreamingAsync(createOptions, cancellationToken)
.AsChatResponseUpdatesAsync(createOptions, cancellationToken)
.ConfigureAwait(false))
{
updates.Add(update);
var reasoningUpdate = update.RawRepresentation
as StreamingResponseReasoningSummaryTextDeltaUpdate;
if (reasoningUpdate is null)
{
continue;
}
var key = (reasoningUpdate.ItemId, reasoningUpdate.SummaryIndex);
if (!reasoningSummaries.TryGetValue(key, out var summary))
{
summary = new StringBuilder();
reasoningSummaries[key] = summary;
}
summary.Append(reasoningUpdate.Delta);
}
return new ResponsesChatResult(
updates.ToChatResponse(),
reasoningSummaries.Values
.Select(summary => summary.ToString())
.Where(summary => !string.IsNullOrWhiteSpace(summary))
.ToArray());
}
private static ResponsesChatResult CreateNonStreamingResult(ChatResponse response)
{
return new ResponsesChatResult(response, ExtractVisibleReasoningSummaries(response));
}
private HttpRequestMessage CreateJsonRequest(
string requestUri,
string payloadJson,
@@ -427,7 +521,9 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
private static bool IsRetryableException(Exception exception)
{
return exception is HttpRequestException or TaskCanceledException;
return exception is ClientResultException clientException
&& IsRetryableStatusCode(clientException.Status)
|| exception is HttpRequestException or TaskCanceledException;
}
private void LogContextWindow(int estimatedTokens, bool compacted, string source)
@@ -499,18 +595,18 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
instructionsPreview);
}
private void LogResponseDiagnostics(string responseJson, ChatResponse response)
private void LogResponseDiagnostics(ChatResponse response)
{
if (logger is null)
{
return;
}
var outputTypes = GetOutputItemTypes(responseJson);
var parsedTypes = response.Messages
.SelectMany(message => message.Contents)
.Select(content => content.GetType().Name)
.ToArray();
var outputTypes = GetOutputItemTypes(response);
var functionCalls = response.Messages
.SelectMany(message => message.Contents)
.OfType<FunctionCallContent>()
@@ -528,40 +624,29 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
if (parsedTypes.Length == 0)
{
logger.LogWarning(
"LiteLLM Responses response contained no parseable message text or function calls. Raw response preview: {ResponsePreview}",
Truncate(responseJson, maxLength: 6000));
"LiteLLM Responses response contained no parseable message text or function calls.");
}
}
private static void ThrowIfResponseHasNoContent(string responseJson, ChatResponse response)
private static void ThrowIfResponseHasNoContent(ChatResponse response)
{
if (response.Messages.SelectMany(message => message.Contents).Any())
{
return;
}
var outputTypes = GetOutputItemTypes(responseJson);
var outputTypes = GetOutputItemTypes(response);
throw new InvalidOperationException(
"LiteLLM Responses returned no parseable message text or function calls. " +
$"ResponseId={response.ResponseId ?? "<none>"}, outputTypes=[{string.Join(", ", outputTypes)}].");
}
private static string[] GetOutputItemTypes(string responseJson)
private static string[] GetOutputItemTypes(ChatResponse response)
{
try
{
using var document = JsonDocument.Parse(responseJson);
return document.RootElement.TryGetProperty("output", out var output) && output.ValueKind == JsonValueKind.Array
? output
.EnumerateArray()
.Select(item => GetString(item, "type") ?? item.ValueKind.ToString())
.ToArray()
: [];
}
catch (JsonException)
{
return ["<invalid-json>"];
}
return response.Messages
.SelectMany(message => message.Contents)
.Select(content => content.RawRepresentation?.GetType().Name ?? content.GetType().Name)
.ToArray();
}
private static string Truncate(string value, int maxLength)
@@ -574,9 +659,9 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
private static void AddInputItem(JsonArray input, StringBuilder instructions, ChatMessage message)
{
var role = message.Role.Value;
var text = message.Text;
if (role == ChatRole.System.Value)
{
var text = message.Text;
if (!string.IsNullOrWhiteSpace(text))
{
instructions.AppendLine(text);
@@ -585,12 +670,37 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
return;
}
if (!string.IsNullOrWhiteSpace(text))
var isAssistant = role == ChatRole.Assistant.Value;
var messageContent = new JsonArray();
foreach (var content in message.Contents)
{
if (content is TextContent textContent && !string.IsNullOrWhiteSpace(textContent.Text))
{
messageContent.Add(new JsonObject
{
["type"] = isAssistant ? "output_text" : "input_text",
["text"] = textContent.Text
});
}
else if (!isAssistant &&
content is DataContent dataContent &&
dataContent.HasTopLevelMediaType("image"))
{
messageContent.Add(new JsonObject
{
["type"] = "input_image",
["image_url"] = dataContent.Uri.ToString()
});
}
}
if (messageContent.Count > 0)
{
input.Add(new JsonObject
{
["role"] = role == ChatRole.Assistant.Value ? "assistant" : "user",
["content"] = text
["type"] = "message",
["role"] = isAssistant ? "assistant" : "user",
["content"] = messageContent
});
}
@@ -603,7 +713,7 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
["type"] = "function_call",
["call_id"] = call.CallId,
["name"] = call.Name,
["arguments"] = JsonSerializer.Serialize(call.Arguments, JsonOptions)
["arguments"] = SerializeFunctionArguments(call)
});
}
else if (content is FunctionResultContent result)
@@ -640,50 +750,15 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
return result;
}
private static void AddMessageContent(List<AIContent> contents, JsonElement item)
private static string SerializeFunctionArguments(FunctionCallContent call)
{
if (!item.TryGetProperty("content", out var messageContent) || messageContent.ValueKind != JsonValueKind.Array)
if (call.RawRepresentation is FunctionCallResponseItem responseItem
&& responseItem.FunctionArguments is not null)
{
return;
return responseItem.FunctionArguments.ToString();
}
foreach (var content in messageContent.EnumerateArray())
{
if (GetString(content, "type") == "output_text")
{
var text = GetString(content, "text");
if (!string.IsNullOrEmpty(text))
{
contents.Add(new TextContent(text));
}
}
}
}
private static Dictionary<string, object?> ParseArguments(string? arguments)
{
if (string.IsNullOrWhiteSpace(arguments))
{
return [];
}
using var document = JsonDocument.Parse(arguments);
return document.RootElement.EnumerateObject()
.ToDictionary(property => property.Name, property => ConvertJsonValue(property.Value));
}
private static object? ConvertJsonValue(JsonElement element)
{
return element.ValueKind switch
{
JsonValueKind.String => element.GetString(),
JsonValueKind.Number when element.TryGetInt64(out var integer) => integer,
JsonValueKind.Number when element.TryGetDouble(out var number) => number,
JsonValueKind.True => true,
JsonValueKind.False => false,
JsonValueKind.Null => null,
_ => JsonSerializer.Deserialize<object>(element.GetRawText(), JsonOptions)
};
return JsonSerializer.Serialize(call.Arguments, JsonOptions);
}
private static string ResultToString(object? result)
@@ -696,58 +771,13 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
};
}
private static UsageDetails? ParseUsage(JsonElement root)
{
if (!root.TryGetProperty("usage", out var usage) || usage.ValueKind != JsonValueKind.Object)
{
return null;
}
var inputTokens = GetInt(usage, "input_tokens") ?? GetInt(usage, "prompt_tokens");
var outputTokens = GetInt(usage, "output_tokens") ?? GetInt(usage, "completion_tokens");
var totalTokens = GetInt(usage, "total_tokens");
return new UsageDetails
{
InputTokenCount = inputTokens,
OutputTokenCount = outputTokens,
TotalTokenCount = totalTokens
};
}
private static int EstimateTokens(JsonObject payload)
{
var json = payload.ToJsonString(JsonOptions);
return Math.Max(1, (int)Math.Ceiling(json.Length / 4.0));
}
private static int? GetInt(JsonElement element, string propertyName)
{
return element.TryGetProperty(propertyName, out var property) && property.TryGetInt32(out var value)
? value
: null;
}
private static string GetRequiredString(JsonElement element, string propertyName)
{
return GetString(element, propertyName)
?? throw new InvalidOperationException($"LiteLLM response item did not include '{propertyName}'.");
}
private static string? GetString(JsonElement element, string propertyName)
{
return element.TryGetProperty(propertyName, out var property) && property.ValueKind == JsonValueKind.String
? property.GetString()
: null;
}
private static DateTimeOffset? GetUnixTimestamp(JsonElement element, string propertyName)
{
return element.TryGetProperty(propertyName, out var property) && property.TryGetInt64(out var value)
? DateTimeOffset.FromUnixTimeSeconds(value)
: null;
}
private static Uri NormalizeEndpoint(Uri endpoint)
internal static Uri NormalizeEndpoint(Uri endpoint)
{
var value = endpoint.ToString().TrimEnd('/');
if (!value.EndsWith("/v1", StringComparison.OrdinalIgnoreCase))
@@ -762,5 +792,60 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
{
return value.TrimStart('/');
}
private static ResponsesClient CreateResponsesClient(
HttpClient httpClient,
string apiKey,
AsyncLocal<string?> requestInitiator)
{
var options = new OpenAIClientOptions
{
Endpoint = httpClient.BaseAddress
?? throw new InvalidOperationException("LiteLLM HTTP client requires a base address."),
Transport = new HttpClientPipelineTransport(httpClient),
RetryPolicy = new ClientRetryPolicy(0)
};
options.AddPolicy(
new InitiatorHeaderPolicy(requestInitiator),
PipelinePosition.PerCall);
return new ResponsesClient(
new ApiKeyCredential(apiKey),
options);
}
private sealed record ResponsesChatResult(
ChatResponse Response,
IReadOnlyList<string> ReasoningSummaries);
private sealed class InitiatorHeaderPolicy(AsyncLocal<string?> requestInitiator) : PipelinePolicy
{
public override void Process(
PipelineMessage message,
IReadOnlyList<PipelinePolicy> pipeline,
int currentIndex)
{
SetHeader(message);
ProcessNext(message, pipeline, currentIndex);
}
public override ValueTask ProcessAsync(
PipelineMessage message,
IReadOnlyList<PipelinePolicy> pipeline,
int currentIndex)
{
SetHeader(message);
return ProcessNextAsync(message, pipeline, currentIndex);
}
private void SetHeader(PipelineMessage message)
{
if (!string.IsNullOrWhiteSpace(requestInitiator.Value))
{
message.Request.Headers.Set("X-Initiator", requestInitiator.Value);
}
}
}
}
#pragma warning restore OPENAI001
#pragma warning restore MAAI001
@@ -14,7 +14,7 @@ public sealed class MeetingSummaryInstructionBuilder : IMeetingSummaryInstructio
write_summary is only for the current meeting's summary file. Past project meeting summaries are read-only historical context; use list_past_project_meetings and read_past_project_meeting_summary to inspect them, and never try to mutate them.
If the meeting note has no title, or still has a generated default title like `Meeting yyyy-MM-dd HH:mm`, provide a concise title parameter to write_summary when the purpose of the meeting is clear from transcript, user notes, or assistant context. If the purpose is not clear, omit the title parameter.
Use read_meetingnote to inspect frontmatter such as title, attendees, projects, start_time, and end_time.
Use read_context and write_context as your own meeting notebook. Its frontmatter may include agenda from the calendar appointment. Record useful internal notes, missing context, requests for future tools, suggested improvements, and relevant context discovered from other sources. write_context appends by default; use replace_file=true only when you are intentionally replacing the whole assistant context body. Keep user-facing summary content in the summary note.
Treat the assistant context as persistent meeting-specific memory and use read_context and write_context as your own meeting notebook. Its frontmatter may include agenda from the calendar appointment. Whenever you encounter unexpected problems, discover missing information, or make assumptions while summarizing, use write_context to append a concise note so later agents working on this meeting can understand them. Also record useful internal notes, requests for future tools, suggested improvements, and relevant context discovered from other sources. write_context appends by default; use replace_file=true only when you are intentionally replacing the whole assistant context body. Keep user-facing summary content in the summary note.
Use add_dictation_word when project context, user notes, or transcript evidence show that a domain term, acronym, name, or unusual word is likely to be repeatedly mistranscribed. Add only the canonical spelling, one term at a time.
Use add_attendee and remove_attendee to sharpen the meeting attendees list from clear transcript evidence and screenshot OCR participant evidence. Treat OCR that says visible people are a partial screenshot result as incomplete evidence; do not remove attendees solely because they are absent from a partial screenshot.
Use override_speaker only when you are very certain that a transcript speaker label belongs to a named person, for example from user notes, OCR evidence with a matching timestamp, or very clear context cues. Provide the exact speaker label from the transcript and the replacement speaker name. If the replacement speaker already exists in the transcript, use merge=true only when you are certain both speaker labels are the same identity; otherwise do not merge them.
@@ -22,7 +22,7 @@ public sealed class MeetingSummaryInstructionBuilder : IMeetingSummaryInstructio
After writing the summary, update existing project files when the meeting produced durable project knowledge, decisions, next steps, or context.
Use list_projects first to see which projects are bound to this meeting. Use search, list_past_project_meetings, read_past_project_meeting_summary, and read_projectfile before changing existing project files. search includes both project files and past meeting summaries for the requested current-meeting project scope.
The summary note should contain concise sections for summary, decisions, open questions, and next steps.
If the assistant context contains cropped screenshot markdown links, include only the most relevant cropped screenshots in the summary by markdown-linking them near the related summary text. Do not include every cropped screenshot by default, and do not link uncropped screenshots unless no cropped version exists and the image is important.
If the assistant context contains cropped screenshot markdown links, include only the most relevant cropped screenshots in the summary by markdown-linking them near the related summary text. When embedding them, encode spaces in image-link targets as `%20`; never leave literal spaces in the link target. Do not include every cropped screenshot by default, and do not link uncropped screenshots unless no cropped version exists and the image is important.
Keep the output grounded in the source material and explicitly say when a section has no known items.
""";
@@ -1,6 +1,7 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
using MeetingAssistant.Workflow;
using YamlDotNet.Serialization;
namespace MeetingAssistant.Summary;
@@ -13,6 +14,7 @@ public sealed class MeetingSummaryTools
private readonly MeetingAssistantOptions options;
private readonly IDictationWordStore? dictationWordStore;
private readonly SummaryAgentWriteAudit? writeAudit;
private readonly IMeetingWorkflowEngine meetingWorkflowEngine;
private readonly BoundMeetingProjectResolver projectResolver;
public MeetingSummaryTools(MeetingSessionArtifacts artifacts)
@@ -24,12 +26,14 @@ public sealed class MeetingSummaryTools
MeetingSessionArtifacts artifacts,
MeetingAssistantOptions options,
IDictationWordStore? dictationWordStore = null,
SummaryAgentWriteAudit? writeAudit = null)
SummaryAgentWriteAudit? writeAudit = null,
IMeetingWorkflowEngine? meetingWorkflowEngine = null)
{
this.artifacts = artifacts;
this.options = options;
this.dictationWordStore = dictationWordStore;
this.writeAudit = writeAudit;
this.meetingWorkflowEngine = meetingWorkflowEngine ?? NoopMeetingWorkflowEngine.Instance;
projectResolver = new BoundMeetingProjectResolver(options);
}
@@ -98,7 +102,11 @@ public sealed class MeetingSummaryTools
return "Refused: attendee must not be empty.";
}
var displayName = attendee.Trim();
var displayName = await meetingWorkflowEngine.TransformAttendeeAsync(
MeetingWorkflowEvent.AttendeeAdded(artifacts, attendee.Trim()),
options,
CancellationToken.None);
displayName = displayName.Trim();
var normalized = MeetingAttendeeNames.NormalizeDisplayName(displayName);
if (string.IsNullOrWhiteSpace(normalized))
{
@@ -1,5 +1,6 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Transcription;
using MeetingAssistant.Workflow;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
@@ -17,6 +18,7 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
private readonly IMeetingSummaryFailureWriter failureWriter;
private readonly IMeetingSummaryInstructionBuilder instructionBuilder;
private readonly IDictationWordStore dictationWordStore;
private readonly IMeetingWorkflowEngine meetingWorkflowEngine;
public OpenAiMeetingSummaryAgentPipeline(
IOptions<MeetingAssistantOptions> options,
@@ -25,7 +27,8 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
IServiceProvider services,
IMeetingSummaryFailureWriter failureWriter,
IMeetingSummaryInstructionBuilder instructionBuilder,
IDictationWordStore dictationWordStore)
IDictationWordStore dictationWordStore,
IMeetingWorkflowEngine meetingWorkflowEngine)
{
this.options = options.Value;
this.loggerFactory = loggerFactory;
@@ -34,6 +37,7 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
this.failureWriter = failureWriter;
this.instructionBuilder = instructionBuilder;
this.dictationWordStore = dictationWordStore;
this.meetingWorkflowEngine = meetingWorkflowEngine;
}
public async Task<MeetingSummaryRunResult> RunAsync(
@@ -51,35 +55,32 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
var agentOptions = options.Agent;
var key = ResolveApiKey(agentOptions);
var writeAudit = new SummaryAgentWriteAudit(artifacts);
var meetingTools = new MeetingSummaryTools(artifacts, options, dictationWordStore, writeAudit);
var meetingTools = new MeetingSummaryTools(
artifacts,
options,
dictationWordStore,
writeAudit,
meetingWorkflowEngine);
var tools = CreateTools(meetingTools);
var instructions = await instructionBuilder.BuildAsync(artifacts, options, cancellationToken);
using var compactionSummaryClient = new LiteLlmResponsesChatClient(
new Uri(agentOptions.Endpoint),
using var compactionSummaryClient = LiteLlmResponsesChatClient.Create(
agentOptions,
key,
agentOptions.Model,
agentOptions.EnableThinking,
ToReasoningEffortValue(agentOptions.ReasoningEffort),
agentOptions.ReconnectionAttempts,
agentOptions.ReconnectionDelay,
compactionOptions: null,
logger,
firstRequestIsUser: false);
var compactionOptions = CreateCompactionOptions(agentOptions, compactionSummaryClient);
using var chatClient = new LiteLlmResponsesChatClient(
new Uri(agentOptions.Endpoint),
using var chatClient = LiteLlmResponsesChatClient.Create(
agentOptions,
key,
agentOptions.Model,
agentOptions.EnableThinking,
ToReasoningEffortValue(agentOptions.ReasoningEffort),
agentOptions.ReconnectionAttempts,
agentOptions.ReconnectionDelay,
compactionOptions,
logger,
firstRequestIsUser: false);
var agent = chatClient
.AsBuilder()
.UseFunctionInvocation()
.UseFunctionInvocation(
loggerFactory,
client => client.FunctionInvoker = FunctionInvocationGuard.InvokeAsync)
.Build()
.AsAIAgent(
instructions,
@@ -303,18 +304,6 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
return new PipelineCompactionStrategy(strategies);
}
private static string ToReasoningEffortValue(ReasoningEffortOption effort)
{
return effort switch
{
ReasoningEffortOption.None => "none",
ReasoningEffortOption.Low => "low",
ReasoningEffortOption.High => "high",
ReasoningEffortOption.ExtraHigh => "xhigh",
_ => "medium"
};
}
private static ReasoningEffort ToReasoningEffort(ReasoningEffortOption effort)
{
return effort switch
+35 -13
View File
@@ -26,7 +26,8 @@ public sealed record MeetingTaskbarMenuItem(
string? ProfileName = null,
string? MicrophoneDeviceId = null,
bool IsChecked = false,
IReadOnlyList<MeetingTaskbarMenuItem>? Items = null);
IReadOnlyList<MeetingTaskbarMenuItem>? Items = null,
bool StartsSection = false);
public static class MeetingTaskbarMenuBuilder
{
@@ -38,26 +39,32 @@ public static class MeetingTaskbarMenuBuilder
{
var items = new List<MeetingTaskbarMenuItem>
{
new("Settings and logs", MeetingTaskbarAction.EditRules)
new("Open agent", MeetingTaskbarAction.EditRules)
};
if (microphones is { Count: > 0 })
{
items.Add(BuildMicrophoneMenu(microphones, currentMicrophoneDeviceId));
}
if (status.IsRecording)
{
items.Add(new MeetingTaskbarMenuItem(
"Stop meeting recording and transcribe",
MeetingTaskbarAction.StopRecording));
items.Add(new MeetingTaskbarMenuItem(
"Finish meeting",
MeetingTaskbarAction.StopRecording,
StartsSection: true));
}
var secondaryControls = new List<MeetingTaskbarMenuItem>();
if (microphones is { Count: > 0 })
{
secondaryControls.Add(BuildMicrophoneMenu(microphones, currentMicrophoneDeviceId));
}
if (status.IsRecording)
{
secondaryControls.Add(new MeetingTaskbarMenuItem(
"Cancel meeting recording and discard",
MeetingTaskbarAction.AbortRecording));
foreach (var profile in launchProfiles.Where(profile => !IsActiveProfile(profile, status)))
{
items.Add(new MeetingTaskbarMenuItem(
secondaryControls.Add(new MeetingTaskbarMenuItem(
AppendHotkey($"Switch to {profile.Name}", profile.Options.Hotkey.Toggle),
MeetingTaskbarAction.SwitchProfile,
profile.Name));
@@ -67,16 +74,18 @@ public static class MeetingTaskbarMenuBuilder
{
foreach (var profile in launchProfiles)
{
items.Add(new MeetingTaskbarMenuItem(
secondaryControls.Add(new MeetingTaskbarMenuItem(
AppendHotkey($"Start meeting recording ({profile.Name})", profile.Options.Hotkey.Toggle),
MeetingTaskbarAction.StartRecording,
profile.Name));
}
}
AddSection(items, secondaryControls);
items.Add(new MeetingTaskbarMenuItem(
"Exit",
MeetingTaskbarAction.Exit));
MeetingTaskbarAction.Exit,
StartsSection: true));
return new MeetingTaskbarMenu(
status.State,
@@ -102,6 +111,19 @@ public static class MeetingTaskbarMenuBuilder
Items: microphoneItems);
}
private static void AddSection(
List<MeetingTaskbarMenuItem> items,
IReadOnlyList<MeetingTaskbarMenuItem> section)
{
if (section.Count == 0)
{
return;
}
items.Add(section[0] with { StartsSection = true });
items.AddRange(section.Skip(1));
}
private static string BuildTooltip(RecordingStatus status)
{
return status.State switch
@@ -196,14 +196,12 @@ public sealed class UnoTaskbarIconService : IHostedService, IDisposable
var popupMenu = new PopupMenu();
for (var index = 0; index < menu.Items.Count; index++)
{
if (index == 1 ||
(menu.Items[index].Action == MeetingTaskbarAction.Exit &&
menu.Items[index - 1].Action != MeetingTaskbarAction.EditRules))
var menuItem = menu.Items[index];
if (index > 0 && menuItem.StartsSection)
{
popupMenu.Items.Add(new PopupMenuSeparator());
}
var menuItem = menu.Items[index];
popupMenu.Items.Add(BuildPopupItem(menuItem));
}
@@ -290,7 +288,7 @@ public sealed class UnoTaskbarIconService : IHostedService, IDisposable
return string.Join(
"|",
FlattenMenuItems(menu.Items).Select(item =>
$"{item.Action}:{item.ProfileName}:{item.MicrophoneDeviceId}:{item.IsChecked}:{item.Text}"));
$"{item.Action}:{item.ProfileName}:{item.MicrophoneDeviceId}:{item.IsChecked}:{item.StartsSection}:{item.Text}"));
}
private static IEnumerable<MeetingTaskbarMenuItem> FlattenMenuItems(
@@ -17,6 +17,14 @@ public interface IMeetingWorkflowEngine
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken);
Task<string> TransformAttendeeAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return Task.FromResult(workflowEvent.AttendeeName ?? "");
}
}
public sealed class NoopMeetingWorkflowEngine : IMeetingWorkflowEngine
@@ -38,6 +46,14 @@ public sealed class NoopMeetingWorkflowEngine : IMeetingWorkflowEngine
{
return Task.FromResult(workflowEvent.TranscriptLineText ?? "");
}
public Task<string> TransformAttendeeAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
return Task.FromResult(workflowEvent.AttendeeName ?? "");
}
}
public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
@@ -54,7 +70,8 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
"state.to",
"speaker.name",
MeetingWorkflowRuleSchema.PropertyTranscriptLine,
MeetingWorkflowRuleSchema.ParameterTranscriptSpeaker
MeetingWorkflowRuleSchema.ParameterTranscriptSpeaker,
MeetingWorkflowRuleSchema.PropertyAttendeeName
];
private readonly IMeetingWorkflowRulesProvider rulesProvider;
@@ -98,6 +115,21 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
?? "";
}
public async Task<string> TransformAttendeeAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
if (workflowEvent.Type != MeetingWorkflowEventType.AttendeeAdded)
{
throw new ArgumentException("Workflow event must be an attendee added event.", nameof(workflowEvent));
}
return await RunCoreAsync(workflowEvent, options, cancellationToken)
?? workflowEvent.AttendeeName
?? "";
}
private async Task<string?> RunCoreAsync(
MeetingWorkflowEvent workflowEvent,
MeetingAssistantOptions options,
@@ -107,7 +139,7 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
var context = new MeetingWorkflowExecutionContext(workflowEvent);
if (rules.Count == 0)
{
return context.TranscriptLine;
return context.ResultValue;
}
var meeting = await meetingNoteStore.ReadAsync(
@@ -134,10 +166,18 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
var model = CreateTemplateModel(meeting, context);
foreach (var step in rule.Steps)
{
if (workflowEvent.Type == MeetingWorkflowEventType.AttendeeAdded &&
!IsAttendeeTransformStep(step))
{
throw new InvalidOperationException(
"attendee_added workflow rules can only transform attendee.name with set_property.");
}
var stepChanged = await ApplyStepAsync(
step,
meeting,
context,
options,
model,
cancellationToken);
ruleNoteChanged |= stepChanged;
@@ -176,7 +216,7 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
cancellationToken);
}
return context.TranscriptLine;
return context.ResultValue;
}
private static bool MatchesTrigger(
@@ -229,9 +269,41 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
StringComparison.OrdinalIgnoreCase));
}
if (trigger.AttendeeAdded is not null)
{
return workflowEvent.Type == MeetingWorkflowEventType.AttendeeAdded &&
MatchesAttendeeAddedTrigger(trigger.AttendeeAdded, workflowEvent.AttendeeName);
}
return false;
}
private static bool MatchesAttendeeAddedTrigger(
MeetingWorkflowAttendeeAddedTrigger trigger,
string? attendeeName)
{
var attendee = attendeeName ?? "";
if (!string.IsNullOrWhiteSpace(trigger.EqualsValue) &&
!string.Equals(attendee, trigger.EqualsValue.Trim(), StringComparison.OrdinalIgnoreCase))
{
return false;
}
if (!string.IsNullOrWhiteSpace(trigger.Contains) &&
!attendee.Contains(trigger.Contains.Trim(), StringComparison.OrdinalIgnoreCase))
{
return false;
}
if (!string.IsNullOrWhiteSpace(trigger.Regex) &&
!Regex.IsMatch(attendee, trigger.Regex, RegexOptions.IgnoreCase))
{
return false;
}
return true;
}
private bool EvaluateConditions(
IReadOnlyList<MeetingWorkflowCondition> conditions,
MeetingNote meeting,
@@ -298,6 +370,7 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
MeetingWorkflowStep step,
MeetingNote meeting,
MeetingWorkflowExecutionContext context,
MeetingAssistantOptions options,
MeetingWorkflowTemplateModel model,
CancellationToken cancellationToken)
{
@@ -305,7 +378,11 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
switch (step.Uses.Trim().ToLowerInvariant())
{
case MeetingWorkflowRuleSchema.StepAddAttendee:
return AddUnique(meeting.Frontmatter.Attendees, value, MeetingAttendeeNames.NormalizeDisplayName);
var attendee = await TransformAttendeeAsync(
MeetingWorkflowEvent.AttendeeAdded(context.Event.Artifacts, value),
options,
cancellationToken);
return AddUnique(meeting.Frontmatter.Attendees, attendee, MeetingAttendeeNames.NormalizeDisplayName);
case MeetingWorkflowRuleSchema.StepRemoveAttendee:
return RemoveValue(meeting.Frontmatter.Attendees, value);
case MeetingWorkflowRuleSchema.StepAddProject:
@@ -319,6 +396,9 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
case MeetingWorkflowRuleSchema.StepSetProperty when IsTranscriptLineProperty(step.Property ?? step.Name):
SetTranscriptLine(context, value);
return false;
case MeetingWorkflowRuleSchema.StepSetProperty when IsAttendeeNameProperty(step.Property ?? step.Name):
SetAttendeeName(context, value);
return false;
case MeetingWorkflowRuleSchema.StepSetProperty:
return SetProperty(meeting, step.Property ?? step.Name, value);
default:
@@ -370,11 +450,39 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
context.TranscriptLine = value;
}
private static void SetAttendeeName(
MeetingWorkflowExecutionContext context,
string value)
{
if (context.Event.Type != MeetingWorkflowEventType.AttendeeAdded)
{
throw new InvalidOperationException("The attendee.name property can only be set during attendee_added events.");
}
if (string.Equals(context.AttendeeName, value, StringComparison.Ordinal))
{
return;
}
context.AttendeeName = value;
}
private static bool IsTranscriptLineProperty(string? property)
{
return MeetingWorkflowRuleSchema.IsProperty(property, MeetingWorkflowRuleSchema.PropertyTranscriptLine);
}
private static bool IsAttendeeNameProperty(string? property)
{
return MeetingWorkflowRuleSchema.IsProperty(property, MeetingWorkflowRuleSchema.PropertyAttendeeName);
}
private static bool IsAttendeeTransformStep(MeetingWorkflowStep step)
{
return MeetingWorkflowRuleSchema.IsStep(step.Uses, MeetingWorkflowRuleSchema.StepSetProperty) &&
IsAttendeeNameProperty(step.Property ?? step.Name);
}
private static bool AddUnique(
List<string> values,
string value,
@@ -456,7 +564,8 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
["state.to"] = workflowEvent.ToState is { } to ? MeetingWorkflowStateNames.ToRuleName(to) : "",
["speaker.name"] = workflowEvent.SpeakerName,
[MeetingWorkflowRuleSchema.PropertyTranscriptLine] = context.TranscriptLine ?? "",
[MeetingWorkflowRuleSchema.ParameterTranscriptSpeaker] = workflowEvent.SpeakerName ?? ""
[MeetingWorkflowRuleSchema.ParameterTranscriptSpeaker] = workflowEvent.SpeakerName ?? "",
[MeetingWorkflowRuleSchema.PropertyAttendeeName] = context.AttendeeName ?? ""
};
}
@@ -480,7 +589,10 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
: new MeetingWorkflowSpeakerModel(workflowEvent.SpeakerName),
new MeetingWorkflowTranscriptModel(
context.TranscriptLine ?? "",
workflowEvent.SpeakerName ?? ""));
workflowEvent.SpeakerName ?? ""),
string.IsNullOrWhiteSpace(context.AttendeeName)
? null
: new MeetingWorkflowAttendeeModel(context.AttendeeName));
}
private sealed class MeetingWorkflowExecutionContext
@@ -489,10 +601,23 @@ public sealed class MeetingWorkflowEngine : IMeetingWorkflowEngine
{
Event = workflowEvent;
TranscriptLine = workflowEvent.TranscriptLineText;
AttendeeName = workflowEvent.AttendeeName;
}
public MeetingWorkflowEvent Event { get; }
public string? TranscriptLine { get; set; }
public string? AttendeeName { get; set; }
public string? ResultValue
{
get
{
return Event.Type == MeetingWorkflowEventType.AttendeeAdded
? AttendeeName
: TranscriptLine;
}
}
}
}
@@ -7,7 +7,8 @@ public enum MeetingWorkflowEventType
Created,
StateTransition,
SpeakerIdentified,
TranscriptLine
TranscriptLine,
AttendeeAdded
}
public sealed record MeetingWorkflowEvent(
@@ -16,7 +17,8 @@ public sealed record MeetingWorkflowEvent(
AssistantContextState? FromState = null,
AssistantContextState? ToState = null,
string? SpeakerName = null,
string? TranscriptLineText = null)
string? TranscriptLineText = null,
string? AttendeeName = null)
{
public static MeetingWorkflowEvent Created(MeetingSessionArtifacts artifacts)
{
@@ -49,4 +51,14 @@ public sealed record MeetingWorkflowEvent(
SpeakerName: speakerName,
TranscriptLineText: line);
}
public static MeetingWorkflowEvent AttendeeAdded(
MeetingSessionArtifacts artifacts,
string attendeeName)
{
return new MeetingWorkflowEvent(
MeetingWorkflowEventType.AttendeeAdded,
artifacts,
AttendeeName: attendeeName);
}
}
@@ -37,6 +37,9 @@ public sealed class MeetingWorkflowTrigger
[YamlMember(Alias = "transcript_line")]
public MeetingWorkflowTranscriptLineTrigger? TranscriptLine { get; set; }
[YamlMember(Alias = "attendee_added")]
public MeetingWorkflowAttendeeAddedTrigger? AttendeeAdded { get; set; }
}
public sealed class MeetingWorkflowStateTransitionTrigger
@@ -60,6 +63,18 @@ public sealed class MeetingWorkflowTranscriptLineTrigger
public string? Speaker { get; set; }
}
public sealed class MeetingWorkflowAttendeeAddedTrigger
{
[YamlMember(Alias = "equals")]
public string? EqualsValue { get; set; }
[YamlMember(Alias = "contains")]
public string? Contains { get; set; }
[YamlMember(Alias = "regex")]
public string? Regex { get; set; }
}
public sealed class MeetingWorkflowCondition
{
[YamlMember(Alias = "condition")]
@@ -94,7 +109,8 @@ public sealed record MeetingWorkflowTemplateModel(
MeetingWorkflowMeetingModel Meeting,
MeetingWorkflowEventModel Event,
MeetingWorkflowSpeakerModel? Speaker,
MeetingWorkflowTranscriptModel? Transcript);
MeetingWorkflowTranscriptModel? Transcript,
MeetingWorkflowAttendeeModel? Attendee);
public sealed record MeetingWorkflowMeetingModel(
string Title,
@@ -111,6 +127,8 @@ public sealed record MeetingWorkflowSpeakerModel(string Name);
public sealed record MeetingWorkflowTranscriptModel(string Line, string Speaker);
public sealed record MeetingWorkflowAttendeeModel(string Name);
internal static class MeetingWorkflowStateNames
{
public static string ToRuleName(AssistantContextState state)
@@ -12,13 +12,15 @@ internal static class MeetingWorkflowRuleSchema
public const string PropertyMeetingTitle = "meeting.title";
public const string PropertyTranscriptLine = "transcript.line";
public const string ParameterTranscriptSpeaker = "transcript.speaker";
public const string PropertyAttendeeName = "attendee.name";
public static readonly string[] SupportedTriggerKeys =
[
"created",
"state_transition",
"speaker_identified",
"transcript_line"
"transcript_line",
"attendee_added"
];
public static readonly string[] SupportedConditionKeys =
@@ -42,7 +44,8 @@ internal static class MeetingWorkflowRuleSchema
[
PropertyTitle,
PropertyMeetingTitle,
PropertyTranscriptLine
PropertyTranscriptLine,
PropertyAttendeeName
];
public static bool IsStep(string? value, string step)
@@ -59,7 +59,8 @@ internal static class MeetingWorkflowRulesValidator
trigger.Created is not null,
trigger.StateTransition is not null,
trigger.SpeakerIdentified is not null,
trigger.TranscriptLine is not null);
trigger.TranscriptLine is not null,
trigger.AttendeeAdded is not null);
if (configuredCount == 0)
{
errors.Add($"{ruleName} on[{triggerIndex + 1}] must contain one supported trigger: {JoinAllowed(MeetingWorkflowRuleSchema.SupportedTriggerKeys)}.");
@@ -137,6 +138,12 @@ internal static class MeetingWorkflowRulesValidator
return;
}
if (HasAttendeeAddedTrigger(rule) &&
!IsAttendeeNameSetPropertyStep(step))
{
errors.Add($"{stepPath} on attendee_added rules can only transform the attendee with 'set_property' property 'attendee.name'.");
}
if (MeetingWorkflowRuleSchema.IsStep(uses, MeetingWorkflowRuleSchema.StepSetProperty))
{
var property = step.Property ?? step.Name;
@@ -153,6 +160,11 @@ internal static class MeetingWorkflowRulesValidator
{
errors.Add($"{stepPath} can set 'transcript.line' only on rules with a transcript_line trigger.");
}
else if (MeetingWorkflowRuleSchema.IsProperty(property, MeetingWorkflowRuleSchema.PropertyAttendeeName) &&
!HasAttendeeAddedTrigger(rule))
{
errors.Add($"{stepPath} can set 'attendee.name' only on rules with an attendee_added trigger.");
}
}
if (step.Value is { } value &&
@@ -181,6 +193,17 @@ internal static class MeetingWorkflowRulesValidator
return rule.On.Any(static trigger => trigger.TranscriptLine is not null);
}
private static bool HasAttendeeAddedTrigger(MeetingWorkflowRule rule)
{
return rule.On.Any(static trigger => trigger.AttendeeAdded is not null);
}
private static bool IsAttendeeNameSetPropertyStep(MeetingWorkflowStep step)
{
return MeetingWorkflowRuleSchema.IsStep(step.Uses, MeetingWorkflowRuleSchema.StepSetProperty) &&
MeetingWorkflowRuleSchema.IsProperty(step.Property ?? step.Name, MeetingWorkflowRuleSchema.PropertyAttendeeName);
}
private static int CountConfigured(params bool[] values)
{
return values.Count(static value => value);
@@ -206,7 +229,8 @@ internal static class MeetingWorkflowRulesValidator
new MeetingWorkflowSpeakerModel("Ada Lovelace"),
new MeetingWorkflowTranscriptModel(
"[00:00:01] Ada Lovelace: Validation line.",
"Ada Lovelace"));
"Ada Lovelace"),
new MeetingWorkflowAttendeeModel("Ada Lovelace"));
}
private static string FormatRazorValidationMessage(Exception exception)
@@ -10,9 +10,66 @@ public sealed record WorkflowRulesEditorChatMessage(
WorkflowRulesEditorChatRole Role,
string Content);
public sealed record WorkflowRulesEditorChatResult(
string Response,
IReadOnlyList<WorkflowRulesEditorChatMessage> Conversation);
public enum WorkflowRulesEditorConversationItemKind
{
Message,
Activity
}
public sealed record WorkflowRulesEditorConversationItem(
WorkflowRulesEditorConversationItemKind Kind,
WorkflowRulesEditorChatMessage? Message,
string Content,
IReadOnlyList<string>? ActivityLines = null)
{
public bool IsExpanded { get; set; }
public static WorkflowRulesEditorConversationItem ChatMessage(WorkflowRulesEditorChatMessage message)
{
return new WorkflowRulesEditorConversationItem(
WorkflowRulesEditorConversationItemKind.Message,
message,
message.Content);
}
public static WorkflowRulesEditorConversationItem Activity(string content, IReadOnlyList<string> activityLines)
{
return new WorkflowRulesEditorConversationItem(
WorkflowRulesEditorConversationItemKind.Activity,
null,
content,
activityLines);
}
}
public sealed record WorkflowRulesEditorChatResult(string Response);
public enum WorkflowRulesEditorActivityKind
{
Status,
ToolCall,
Thinking
}
public sealed record WorkflowRulesEditorActivityUpdate(
WorkflowRulesEditorActivityKind Kind,
string Text)
{
public static WorkflowRulesEditorActivityUpdate Status(string text)
{
return new WorkflowRulesEditorActivityUpdate(WorkflowRulesEditorActivityKind.Status, text);
}
public static WorkflowRulesEditorActivityUpdate ToolCall(string toolName)
{
return new WorkflowRulesEditorActivityUpdate(WorkflowRulesEditorActivityKind.ToolCall, toolName);
}
public static WorkflowRulesEditorActivityUpdate Thinking(string text)
{
return new WorkflowRulesEditorActivityUpdate(WorkflowRulesEditorActivityKind.Thinking, text);
}
}
public interface IWorkflowRulesEditorChatPipeline
{
@@ -20,5 +77,5 @@ public interface IWorkflowRulesEditorChatPipeline
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<string>? statusChanged = null);
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null);
}
@@ -59,11 +59,11 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
IReadOnlyList<WorkflowRulesEditorChatMessage> conversation,
string userMessage,
CancellationToken cancellationToken,
Action<string>? statusChanged = null)
Action<WorkflowRulesEditorActivityUpdate>? activityChanged = null)
{
if (string.IsNullOrWhiteSpace(userMessage))
{
return new WorkflowRulesEditorChatResult("", conversation);
return new WorkflowRulesEditorChatResult("");
}
var agentOptions = options.WorkflowRulesEditor.ToEffectiveAgentOptions(options.Agent);
@@ -85,33 +85,35 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
.ToList();
var instructions = await instructionBuilder.BuildAsync(options, cancellationToken);
using var compactionSummaryClient = new LiteLlmResponsesChatClient(
new Uri(agentOptions.Endpoint),
using var compactionSummaryClient = LiteLlmResponsesChatClient.Create(
agentOptions,
key,
agentOptions.Model,
agentOptions.EnableThinking,
ToReasoningEffortValue(agentOptions.ReasoningEffort),
agentOptions.ReconnectionAttempts,
agentOptions.ReconnectionDelay,
compactionOptions: null,
logger,
firstRequestIsUser: false);
var compactionOptions = CreateCompactionOptions(agentOptions, compactionSummaryClient);
using var chatClient = new LiteLlmResponsesChatClient(
new Uri(agentOptions.Endpoint),
using var chatClient = LiteLlmResponsesChatClient.Create(
agentOptions,
key,
agentOptions.Model,
agentOptions.EnableThinking,
ToReasoningEffortValue(agentOptions.ReasoningEffort),
agentOptions.ReconnectionAttempts,
agentOptions.ReconnectionDelay,
compactionOptions,
logger,
firstRequestIsUser: true,
retrying: () => statusChanged?.Invoke("Reconnecting..."));
retrying: () => activityChanged?.Invoke(WorkflowRulesEditorActivityUpdate.Status("Reconnecting...")),
reasoningSummaryChanged: text => activityChanged?.Invoke(WorkflowRulesEditorActivityUpdate.Thinking(text)));
var functionClient = chatClient
.AsBuilder()
.UseFunctionInvocation(loggerFactory)
.UseFunctionInvocation(loggerFactory, client =>
{
client.FunctionInvoker = async (context, token) =>
{
if (context.CallContent.Exception is null)
{
activityChanged?.Invoke(WorkflowRulesEditorActivityUpdate.ToolCall(context.Function.Name));
}
return await FunctionInvocationGuard.InvokeAsync(context, token);
};
})
.Build();
var response = await functionClient.GetResponseAsync(
@@ -121,11 +123,7 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
var responseText = string.IsNullOrWhiteSpace(response.Text)
? "(No response text returned.)"
: response.Text.Trim();
var nextConversation = conversation
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, userMessage.Trim()))
.Append(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.Agent, responseText))
.ToList();
return new WorkflowRulesEditorChatResult(responseText, nextConversation);
return new WorkflowRulesEditorChatResult(responseText);
}
private static ChatMessage ToChatMessage(WorkflowRulesEditorChatMessage message)
@@ -392,18 +390,6 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
$"No {agentName} API key configured. Set MeetingAssistant:WorkflowRulesEditor:Key, MeetingAssistant:Agent:Key, or environment variable '{options.KeyEnv}'.");
}
private static string ToReasoningEffortValue(ReasoningEffortOption effort)
{
return effort switch
{
ReasoningEffortOption.None => "none",
ReasoningEffortOption.Low => "low",
ReasoningEffortOption.High => "high",
ReasoningEffortOption.ExtraHigh => "xhigh",
_ => "medium"
};
}
private static ReasoningEffort ToReasoningEffort(ReasoningEffortOption effort)
{
return effort switch
@@ -1,9 +1,11 @@
using System.Collections.ObjectModel;
using System.Diagnostics;
namespace MeetingAssistant.Workflow;
public sealed class WorkflowRulesEditorChatViewModel
{
private const string ThinkingPlaceholder = "Thinking...";
private readonly IWorkflowRulesEditorChatPipeline pipeline;
public WorkflowRulesEditorChatViewModel(IWorkflowRulesEditorChatPipeline pipeline)
@@ -11,13 +13,15 @@ public sealed class WorkflowRulesEditorChatViewModel
this.pipeline = pipeline;
}
public ObservableCollection<WorkflowRulesEditorChatMessage> Messages { get; } = [];
public ObservableCollection<WorkflowRulesEditorConversationItem> Messages { get; } = [];
public ObservableCollection<string> ActivityMessages { get; } = [];
public string Draft { get; set; } = "";
public bool IsThinking { get; private set; }
public string ActivityMessage { get; private set; } = "Thinking...";
public string ActivityMessage { get; private set; } = ThinkingPlaceholder;
public event EventHandler? Changed;
@@ -30,10 +34,19 @@ public sealed class WorkflowRulesEditorChatViewModel
}
Draft = "";
var priorConversation = Messages.ToList();
Messages.Add(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, prompt));
var startedAt = Stopwatch.GetTimestamp();
var activityLines = new List<string>();
var hasVisibleThinking = false;
var activityContext = SynchronizationContext.Current;
var priorConversation = Messages
.Select(item => item.Message)
.OfType<WorkflowRulesEditorChatMessage>()
.ToList();
Messages.Add(WorkflowRulesEditorConversationItem.ChatMessage(
new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, prompt)));
ActivityMessages.Clear();
IsThinking = true;
ActivityMessage = "Thinking...";
ActivityMessage = ThinkingPlaceholder;
OnChanged();
try
@@ -42,36 +55,121 @@ public sealed class WorkflowRulesEditorChatViewModel
priorConversation,
prompt,
cancellationToken,
SetActivityMessage);
Messages.Clear();
foreach (var message in result.Conversation)
{
Messages.Add(message);
}
update => hasVisibleThinking |= ApplyActivityUpdate(activityContext, update, activityLines));
AddCompletedActivity(startedAt, activityLines, hasVisibleThinking);
Messages.Add(WorkflowRulesEditorConversationItem.ChatMessage(
new WorkflowRulesEditorChatMessage(
WorkflowRulesEditorChatRole.Agent,
string.IsNullOrWhiteSpace(result.Response)
? "(No response text returned.)"
: result.Response.Trim())));
}
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
Messages.Add(new WorkflowRulesEditorChatMessage(
WorkflowRulesEditorChatRole.Agent,
$"Settings and logs failed: {exception.Message}"));
AddCompletedActivity(startedAt, activityLines, hasVisibleThinking);
Messages.Add(WorkflowRulesEditorConversationItem.ChatMessage(
new WorkflowRulesEditorChatMessage(
WorkflowRulesEditorChatRole.Agent,
$"Meeting Summary Agent failed: {exception.Message}")));
}
finally
{
IsThinking = false;
ActivityMessage = "Thinking...";
ActivityMessages.Clear();
ActivityMessage = ThinkingPlaceholder;
OnChanged();
}
}
private void SetActivityMessage(string message)
private bool ApplyActivityUpdate(
SynchronizationContext? context,
WorkflowRulesEditorActivityUpdate update,
List<string> activityLines)
{
if (!IsThinking || string.IsNullOrWhiteSpace(message))
if (context is null || SynchronizationContext.Current == context)
{
return;
return ApplyActivityUpdate(update, activityLines);
}
var hasVisibleThinking = false;
Exception? exception = null;
context.Send(
_ =>
{
try
{
hasVisibleThinking = ApplyActivityUpdate(update, activityLines);
}
catch (Exception caught)
{
exception = caught;
}
},
null);
if (exception is not null)
{
throw exception;
}
return hasVisibleThinking;
}
private bool ApplyActivityUpdate(
WorkflowRulesEditorActivityUpdate update,
List<string> activityLines)
{
if (!IsThinking || string.IsNullOrWhiteSpace(update.Text))
{
return false;
}
var hasVisibleThinking = false;
if (update.Kind == WorkflowRulesEditorActivityKind.ToolCall)
{
var line = $"Called tool: {update.Text.Trim()}";
ActivityMessages.Add(line);
activityLines.Add(line);
}
else if (update.Kind == WorkflowRulesEditorActivityKind.Thinking)
{
var line = update.Text.Trim();
ActivityMessages.Add(line);
activityLines.Add(line);
hasVisibleThinking = true;
}
else
{
var line = update.Text.Trim();
ActivityMessage = line;
activityLines.Add(line);
}
ActivityMessage = message;
OnChanged();
return hasVisibleThinking;
}
private void AddCompletedActivity(
long startedAt,
IReadOnlyList<string> activityLines,
bool hasVisibleThinking)
{
var completedActivityLines = hasVisibleThinking
? activityLines.ToArray()
: new[] { ThinkingPlaceholder }.Concat(activityLines).ToArray();
Messages.Add(WorkflowRulesEditorConversationItem.Activity(
$"Worked for {FormatDuration(Stopwatch.GetElapsedTime(startedAt))}",
completedActivityLines));
}
private static string FormatDuration(TimeSpan duration)
{
if (duration.TotalMinutes >= 1)
{
return $"{(int)duration.TotalMinutes}m {duration.Seconds}s";
}
return $"{Math.Max(0, (int)Math.Round(duration.TotalSeconds))}s";
}
private void OnChanged()
@@ -60,7 +60,7 @@ public sealed class WorkflowRulesEditorInstructionBuilder : IWorkflowRulesEditor
"Log tools can read and search the current application-owned log file and four rotated older files under the temp log folder." + Environment.NewLine + Environment.NewLine +
"Spec tools can search and read copied OpenSpec markdown files from openspec/specs." + Environment.NewLine + Environment.NewLine +
"Project tools can create project folders and read, write, list, and search files in configured projects." + Environment.NewLine + Environment.NewLine +
"Meeting artifact tools can list recent summaries and read/search/write summaries, transcripts, meeting notes, and assistant context files for note post-processing and repair. Prefer frontmatter-specific write tools for metadata-only fixes." + Environment.NewLine + Environment.NewLine +
"Meeting artifact tools can list recent summaries and read/search/write summaries, transcripts, meeting notes, and assistant context files for note post-processing and repair. Prefer frontmatter-specific write tools for metadata-only fixes. Treat each assistant context file as meeting-specific memory. When asked to fix or investigate a meeting or summary, read the matching assistant context for clues about problems, missing information, assumptions, prior fixes, and conclusions. After completing repairs, use write_context to append a concise record of your fixes and conclusions to that assistant context for future work on the meeting." + Environment.NewLine + Environment.NewLine +
"Diagnostic tools mirror the local health, recording status, Outlook metadata, speaker identity merge, workflow reload, and ASR diagnostic endpoints without requiring HTTP." + Environment.NewLine + Environment.NewLine +
"Speaker identity tools can search/list/read/update/delete/merge identities, refuse sampleless identity creation, list/read/delete identity samples, and queue a sample for local playback. Do not delete the last sample from an identity; delete the identity instead." + Environment.NewLine + Environment.NewLine +
"Workflow rules reference documentation:" + Environment.NewLine +
@@ -11,7 +11,7 @@ namespace MeetingAssistant.Workflow;
internal sealed class WpfWorkflowRulesEditorWindowService : IWorkflowRulesEditorWindowService
{
internal const string WindowTitle = "Settings and logs";
internal const string WindowTitle = "Meeting Summary Agent";
private readonly IServiceProvider services;
private readonly WorkflowRulesEditorMarkdownLinkResolver linkResolver;
@@ -288,19 +288,21 @@ internal sealed class WorkflowRulesEditorWpfWindow : Window
conversationPanel.ActualHeight);
conversationPanel.Children.Clear();
foreach (var message in viewModel.Messages)
foreach (var item in viewModel.Messages)
{
conversationPanel.Children.Add(CreateMessageCard(message));
conversationPanel.Children.Add(item.Kind == WorkflowRulesEditorConversationItemKind.Activity
? CreateActivityExpander(item)
: CreateMessageCard(item.Message!));
}
if (viewModel.IsThinking)
{
conversationPanel.Children.Add(new TextBlock
foreach (var activity in viewModel.ActivityMessages)
{
Text = viewModel.ActivityMessage,
Foreground = MutedText,
Margin = new Thickness(4, 2, 4, 2)
});
conversationPanel.Children.Add(CreateActivityLine(activity));
}
conversationPanel.Children.Add(CreateActivityLine(viewModel.ActivityMessage));
}
sendButton.IsEnabled = !viewModel.IsThinking;
@@ -338,6 +340,43 @@ internal sealed class WorkflowRulesEditorWpfWindow : Window
return card;
}
private static Expander CreateActivityExpander(WorkflowRulesEditorConversationItem item)
{
var details = new StackPanel
{
Orientation = Orientation.Vertical,
Margin = new Thickness(16, 2, 4, 6)
};
foreach (var line in item.ActivityLines ?? [])
{
details.Children.Add(CreateActivityLine(line));
}
var expander = new Expander
{
Header = item.Content,
Content = details,
IsExpanded = item.IsExpanded,
Foreground = MutedText,
Background = Brushes.Transparent,
Margin = new Thickness(4, 0, 4, 8)
};
expander.Expanded += (_, _) => item.IsExpanded = true;
expander.Collapsed += (_, _) => item.IsExpanded = false;
return expander;
}
private static TextBlock CreateActivityLine(string text)
{
return new TextBlock
{
Text = text,
Foreground = MutedText,
Margin = new Thickness(4, 2, 4, 2)
};
}
private static Style CreateSendButtonStyle()
{
var style = new Style(typeof(Button));
+1
View File
@@ -172,6 +172,7 @@
"Endpoint": "http://127.0.0.1:4021",
"KeyEnv": "LITELLM_API_KEY",
"Model": "chatgpt/gpt-5.5",
"UseStreaming": true,
"EnableThinking": true,
"ReasoningEffort": "Medium",
"ReconnectionAttempts": 2,
+4 -4
View File
@@ -108,7 +108,7 @@ Important settings:
- `CalendarRecordingPrompts`: enables Outlook Classic Teams-start prompts on Windows.
- `Screenshots`: controls the capture hotkey, attachment folder, and optional OCR/vision model.
- `Agent`: configures the OpenAI-compatible summary/project agent endpoint, model, retries, output limits, and compaction.
- `WorkflowRulesEditor`: optionally overrides the agent settings for the tray-launched settings/logs assistant.
- `WorkflowRulesEditor`: optionally overrides the agent settings for the tray-launched assistant.
Required or commonly used secrets:
@@ -125,14 +125,14 @@ See `docs/meeting-assistant-configuration.md` for the full configuration referen
- **Azure AI Speech**: default checked-in ASR path, live diarized conversation transcription, and speaker identity matching.
- **FunASR**: optional WebSocket streaming ASR. When managed backend startup is enabled, the app pulls and runs the configured Docker image as `meeting-assistant-funasr` on port `10095`.
- **Whisper.NET plus pyannote**: optional local Whisper fallback and Docker-backed final diarization.
- **LiteLLM/OpenAI-compatible Responses endpoint**: summary generation, screenshot OCR fallback, project tools, settings/logs assistant, and retry flows.
- **LiteLLM/OpenAI-compatible Responses endpoint**: summary generation, screenshot OCR fallback, project tools, the tray-launched assistant, and retry flows.
- **Docker Desktop or compatible Docker CLI**: required only for managed FunASR and pyannote paths.
## Workflow Rules And Agents
Meeting-specific automation lives in a local YAML file, not in committed personal rules. Rules can trigger on meeting creation, assistant-context state transitions, identified speakers, and transcript-line writes. They can add/remove attendees, set supported properties, add context, add projects, and rewrite a transcript line before persistence.
Meeting-specific automation lives in a local YAML file, not in committed personal rules. Rules can trigger on meeting creation, assistant-context state transitions, identified speakers, transcript-line writes, and added attendees. They can add/remove attendees, set supported properties, add context, add projects, rewrite a transcript line, and transform an attendee name before it is stored.
The tray menu exposes the settings/logs assistant. It can edit workflow rules with validation, inspect logs and health/status, manage speaker identities and samples, run ASR diagnostics, and read/write scoped meeting/project artifacts through explicit tools.
The tray menu exposes `Open agent`, which opens the `Meeting Summary Agent` window. It can edit workflow rules with validation, inspect logs and health/status, manage speaker identities and samples, run ASR diagnostics, and read/write scoped meeting/project artifacts through explicit tools.
Detailed workflow syntax and extension guidance live in `docs/meeting-workflow-engine.md`.
+9 -4
View File
@@ -317,7 +317,7 @@ When enabled on Windows, Meeting Assistant periodically syncs today's Outlook Cl
`Screenshots:Hotkey` configures a global hotkey that captures the currently active window during an active meeting. Screenshots are written under `Screenshots:AttachmentsFolder`, which defaults to an `Attachments` folder beside the assistant context note, and each capture appends a timestamped markdown image link to the assistant context.
`Screenshots:Ocr` optionally enables vision extraction for screenshots. Blank `Endpoint` or `Model` values fall back to the summary `Agent` endpoint and model. `Key` or `KeyEnv` can be set specifically for OCR; otherwise the summary agent key configuration is used. Automatic summarization scans the meeting note for user-added Obsidian image embeds such as `![[whiteboard.png]]` and Markdown image embeds such as `![Diagram](attachments/diagram.png)`, adds resolvable local images to the assistant context without copying them or changing the meeting note, runs OCR without crop or attendee updates, and waits for all pending OCR work to complete or hit `Timeout` before the assistant context moves to `summarizing`. Failed or timed-out screenshot OCR writes a retry link that targets `/meetings/screenshot-ocr/retry` with the exact saved screenshot and OCR block id.
`Screenshots:Ocr` optionally enables vision extraction for screenshots. Blank `Endpoint` or `Model` values fall back to the summary `Agent` endpoint and model. `Key` or `KeyEnv` can be set specifically for OCR; otherwise the summary agent key configuration is used. Screenshot OCR also inherits `Agent:UseStreaming`, using the Responses SSE transport when it is `true` and the non-streaming Responses transport when it is `false`. Automatic summarization scans the meeting note for user-added Obsidian image embeds such as `![[whiteboard.png]]` and Markdown image embeds such as `![Diagram](attachments/diagram.png)`, adds resolvable local images to the assistant context without copying them or changing the meeting note, runs OCR without crop or attendee updates, and waits for all pending OCR work to complete or hit `Timeout` before the assistant context moves to `summarizing`. Failed or timed-out screenshot OCR writes a retry link that targets `/meetings/screenshot-ocr/retry` with the exact saved screenshot and OCR block id.
| Setting | Purpose |
| --- | --- |
@@ -339,6 +339,7 @@ When enabled on Windows, Meeting Assistant periodically syncs today's Outlook Cl
| `Key` | Optional inline API key. Prefer `KeyEnv`. |
| `KeyEnv` | Environment variable name for the agent API key. |
| `Model` | Model id sent to the endpoint. |
| `UseStreaming` | Uses the Responses SSE transport when `true` (the default); uses the non-streaming Responses transport when `false`. |
| `EnableThinking` | Enables reasoning options in requests when supported by the model/backend. |
| `ReasoningEffort` | Reasoning effort: `None`, `Low`, `Medium`, `High`, or `ExtraHigh`. |
| `ReconnectionAttempts` | Retry count for transient model endpoint failures. |
@@ -352,18 +353,22 @@ When enabled on Windows, Meeting Assistant periodically syncs today's Outlook Cl
After transcription has fully finished, Meeting Assistant automatically runs the summary pipeline for the meeting. The summary agent writes the full markdown summary through `write_summary` and must provide a required `oneliner` value, which is stored in summary frontmatter and must not contain line breaks.
The built-in summary-agent instructions treat assistant context as persistent meeting-specific memory. When the agent encounters an unexpected problem, missing information, or an assumption while summarizing, it appends a concise note through `write_context` so later work on the same meeting can use that history. A configured `Agent:InitialPrompt` completely replaces the built-in instructions, so custom prompts must include equivalent guidance when this behavior is desired.
The summary agent can add and remove meeting-note attendees when transcript or OCR evidence is clear. It can override transcript speaker labels only when the evidence is very certain, and it can delete wrongfully matched identities. Final speaker identity learning and candidate updates run after the summary pipeline finishes so they use the summary-refined attendee list and any recorded speaker identity changes.
`ContextWindowTokens`, `MaxOutputTokens`, `EnableCompaction`, and `CompactionRemainingRatio` configure summary-agent context monitoring. Meeting Assistant estimates the outgoing Responses payload size, logs the estimated token count and remaining context, and compacts the conversation when only the configured remaining ratio is left.
## Workflow Rules Editor
`WorkflowRulesEditor` configures the tray-launched settings/logs assistant. Blank values inherit from `Agent`, so it uses the summarizer endpoint, key, model, reasoning, retry, output, and compaction settings unless explicitly overridden.
`WorkflowRulesEditor` configures the tray-launched `Meeting Summary Agent` window. Blank values inherit from `Agent`, so it uses the summarizer endpoint, key, model, Responses transport, reasoning, retry, output, and compaction settings unless explicitly overridden.
The overridable fields are `Endpoint`, `Key`, `KeyEnv`, `Model`, `EnableThinking`, `ReasoningEffort`, `ReconnectionAttempts`, `ReconnectionDelay`, `ContextWindowTokens`, `MaxOutputTokens`, `EnableCompaction`, `CompactionRemainingRatio`, `ResponsesCompactPath`, and `InitialPrompt`.
The overridable fields are `Endpoint`, `Key`, `KeyEnv`, `Model`, `UseStreaming`, `EnableThinking`, `ReasoningEffort`, `ReconnectionAttempts`, `ReconnectionDelay`, `ContextWindowTokens`, `MaxOutputTokens`, `EnableCompaction`, `CompactionRemainingRatio`, `ResponsesCompactPath`, and `InitialPrompt`.
The assistant can edit workflow rules, manage speaker identities, read and replace the local appsettings file, read this configuration document, search and read copied OpenSpec specs, inspect application logs, create/search/read/write project files, and post-process past meeting artifacts by listing recent summaries and reading, writing, or searching summaries, transcripts, meeting notes, and assistant context files. For artifact metadata repairs, it has frontmatter-specific write tools that preserve the markdown body.
When asked to fix or investigate a meeting or summary, the agent is instructed to treat the matching assistant context as meeting-specific memory and read it for clues about problems, missing information, assumptions, prior fixes, and conclusions. After completing a repair, it appends a concise record of its fixes and conclusions to that context for future work on the meeting. This meeting-memory guidance is appended to the effective interactive-agent prompt even when `WorkflowRulesEditor:InitialPrompt` is configured.
It also has in-process diagnostic tools that mirror the local HTTP diagnostics for health, recording status, current Outlook meeting lookup, recent speaker identity merging, workflow configuration reload, and ASR transcribe/diarize checks.
The `search_spec` and `read_spec_file` tools are scoped to the copied `openspec/specs` markdown tree. Spec files are copied into build and publish output through an MSBuild glob, so newly added folders under `openspec/specs` are included automatically.
@@ -378,7 +383,7 @@ Independently from stdout and stderr redirection, Meeting Assistant writes an ap
%TEMP%\MeetingAssistant\Logs\meeting-assistant.log
```
On startup it rotates the previous current file to `meeting-assistant.log.1` and keeps up to `.4`. The settings/logs assistant reads and searches these current and rotated files. If another app instance or test host still has the file open, rotation is skipped and logging appends to the current file so startup is not blocked.
On startup it rotates the previous current file to `meeting-assistant.log.1` and keeps up to `.4`. The tray-launched assistant reads and searches these current and rotated files. If another app instance or test host still has the file open, rotation is skipped and logging appends to the current file so startup is not blocked.
## API
+54 -2
View File
@@ -22,7 +22,7 @@ The rules file path is configured through `MeetingAssistant:Automation:RulesPath
If the configured path is empty, missing, or points to a blank file, the workflow engine does nothing.
The tray menu includes `Settings and logs`, which opens a small MewUI chat assistant for this configured rules file, the local speaker identity database, appsettings configuration, and application logs. The assistant uses the summarizer agent configuration by default and can be overridden through `MeetingAssistant:WorkflowRulesEditor`.
The tray menu includes `Open agent`, which opens the `Meeting Summary Agent` chat window for this configured rules file, the local speaker identity database, appsettings configuration, and application logs. The assistant uses the summarizer agent configuration by default and can be overridden through `MeetingAssistant:WorkflowRulesEditor`.
```json
{
@@ -31,6 +31,7 @@ The tray menu includes `Settings and logs`, which opens a small MewUI chat assis
"Endpoint": "",
"KeyEnv": "",
"Model": "",
"UseStreaming": null,
"EnableThinking": null,
"ReasoningEffort": null,
"MaxOutputTokens": null,
@@ -45,6 +46,8 @@ The tray menu includes `Settings and logs`, which opens a small MewUI chat assis
Blank editor values inherit from `MeetingAssistant:Agent`. This keeps the editor on the same LiteLLM Responses endpoint and model as the summarizer unless a value is explicitly configured. Each user-submitted editor turn sends its first model request with `X-Initiator: user`; follow-up model requests in the same turn, such as tool-call continuations, use `X-Initiator: agent`.
While the agent is working, the window shows visible reasoning summaries returned by the Responses endpoint and called tool names as lightweight activity lines above the bottom activity line. After the turn completes, those lines move into a collapsible `Worked for <duration>` expander between the user and assistant messages.
The editor agent prompt includes this document and is restricted to these tools:
- `read_rules`: read the configured rules file, optionally by inclusive 1-based line range.
@@ -69,6 +72,12 @@ The engine runs when `MeetingRecordingCoordinator` emits a meeting workflow even
- `state_transition`: after the assistant context state moves forward.
- `speaker_identified`: when live or final speaker identification reports a display name.
- `transcript_line`: after a formatted live transcript line is durably appended, before any changed line is rewritten in place, and before lines are used in full transcript rewrites.
- `attendee_added`: before a newly added attendee is stored in the meeting note.
`attendee_added` is emitted only by Meeting Assistant code paths that intentionally add an attendee
through the workflow engine, notification actions, OCR/screenshot processing, summarizer tools, or
other attendee-add operations. Direct edits to a meeting note file, including direct artifact/frontmatter
repair writes by an agent, are left verbatim and do not trigger attendee automation.
When a recording is started by accepting a calendar notification, the cached appointment metadata is applied before the `created` event. Rules still receive the normal `created` event and the normal `collecting metadata` to `transcribing` state transition, but they see the accepted appointment title, attendees, agenda, and scheduled end instead of a generated placeholder or a separate Outlook current-meeting lookup result.
@@ -160,6 +169,22 @@ on:
speaker: Guest-1
```
### `attendee_added`
Runs before a newly added attendee is stored. `equals`, `contains`, and `regex` filters are optional; omitted filters match any attendee. `equals` and `contains` are case-insensitive.
```yaml
on:
- attendee_added:
contains: Contoso
```
```yaml
on:
- attendee_added:
regex: '@contoso\.com>$'
```
## Conditions
Conditions use NCalc expressions. Dotted workflow variables may be written directly; the engine rewrites them into NCalc parameters internally.
@@ -182,6 +207,7 @@ Available condition variables:
- `speaker.name`
- `transcript.line`
- `transcript.speaker`
- `attendee.name`
Available helper functions:
@@ -215,6 +241,7 @@ Step `value` fields can be plain strings or Razor templates. Razor templates rec
- `Model.Speaker.Name`
- `Model.Transcript.Line`
- `Model.Transcript.Speaker`
- `Model.Attendee.Name`
Example:
@@ -257,7 +284,11 @@ steps:
### `set_property`
Sets a supported property. Supported meeting properties are `title` and `meeting.title`. During `transcript_line` events, `transcript.line` is also supported. For live transcription, changing `transcript.line` rewrites the referenced formatted line after the workflow task completes.
Sets a supported property. Supported meeting properties are `title` and `meeting.title`. During `transcript_line` events, `transcript.line` is also supported. During `attendee_added` events, `attendee.name` is supported. For live transcription, changing `transcript.line` rewrites the referenced formatted line after the workflow task completes. For attendee additions, changing `attendee.name` stores the transformed attendee instead of the original added value.
Rules triggered by `attendee_added` are transformation-only: their steps must use `set_property`
with `property: attendee.name`. They can still use trigger filters, conditions, and Razor templates
to decide and compute the transformed value, but they cannot perform note/context side effects.
```yaml
steps:
@@ -273,6 +304,13 @@ steps:
value: '@Model.Transcript.Line.Replace("*****", "[redacted]")'
```
```yaml
steps:
- uses: set_property
property: attendee.name
value: '@Model.Attendee.Name.Replace(" (Contoso)", "")'
```
### `add_context`
Appends rendered text to the assistant context artifact body. This step does not count as a meeting-note mutation and does not cause a meeting-note save by itself.
@@ -368,6 +406,20 @@ rules:
value: '@Model.Transcript.Line.Replace("*****", "[redacted]")'
```
Clean attendee names as they are added:
```yaml
rules:
- name: clean-contoso-attendees
on:
- attendee_added:
contains: Contoso
steps:
- uses: set_property
property: attendee.name
value: '@Model.Attendee.Name.Replace(" (Contoso)", "")'
```
## Implementation Notes
Core files:
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-29
@@ -0,0 +1,34 @@
## Context
`LiteLlmScreenshotOcrClient` currently builds and posts a raw Responses JSON payload, then parses the successful body as one JSON document. It inherits endpoint, model, and key values from `AgentOptions`, but never reads `AgentOptions.UseStreaming`. The summary and workflow agents already use `LiteLlmResponsesChatClient`, which selects the OpenAI SDK streaming or non-streaming Responses method and maps both through the Microsoft.Extensions.AI adapter.
## Goals / Non-Goals
**Goals:**
- Make screenshot OCR use `Agent:UseStreaming` without adding another setting.
- Reuse the supported Responses SDK transport and response adapter.
- Preserve screenshot-specific endpoint, model, key, prompt, image, crop, attendee, and timeout behavior.
- Keep non-streaming screenshot OCR working when streaming is disabled.
**Non-Goals:**
- Expose OCR token deltas to the UI or assistant context.
- Change screenshot OCR retry, crop, attendee, or note-block semantics.
- Add file upload or remote image URL support.
## Decisions
1. Route screenshot OCR through `LiteLlmResponsesChatClient` instead of maintaining a second Responses parser. The screenshot client will construct one user chat message containing prompt text and PNG `DataContent`, then consume the buffered `ChatResponse.Text`. This keeps transport selection, SDK request creation, SSE assembly, and non-streaming mapping in one client.
2. Extend the shared Responses message translator to map image `DataContent` to an `input_image` content block using its data URI. Text and image blocks remain in one `type: message` input item, matching the existing OCR payload shape.
3. Use `AgentOptions.UseStreaming` for screenshot OCR even when the screenshot-specific endpoint or model overrides are set. Endpoint/model/key remain independently overrideable; transport is an agent-wide behavior setting.
4. Disable reasoning and compaction for the one-turn OCR request, preserving the existing screenshot client behavior while still using the agent reconnection settings and selected transport.
## Risks / Trade-offs
- **[Shared-client diagnostics mention summary context]** Some low-level logs are named for the summary pipeline. → Avoid passing summary compaction state and keep screenshot-specific completion/failure logs at the screenshot client boundary.
- **[Multimodal translation expands shared client scope]** Incorrect content mapping could affect summary requests. → Add request-body behavior coverage proving prompt and PNG data URI are preserved, while existing summary message tests protect text translation.
- **[Provider image support varies]** A configured model may reject image input. → Preserve the provider error and existing screenshot OCR failure/retry behavior.
@@ -0,0 +1,24 @@
## Why
Screenshot OCR inherits its endpoint, model, and key from the summary agent, but it bypasses the shared Responses client and ignores `Agent:UseStreaming`. With streaming enabled, the screenshot client still expects one JSON document and cannot consume the configured LiteLLM Responses event stream.
## What Changes
- Make screenshot OCR inherit the summary agent's streaming transport selection.
- Send screenshot image input through the same OpenAI Responses SDK and Agent Framework adapter used by the summary client.
- Preserve the existing non-streaming screenshot OCR path when streaming is disabled.
- Add behavior coverage for streamed and non-streamed screenshot OCR responses.
## Capabilities
### New Capabilities
- None.
### Modified Capabilities
- `meeting-summary`: Require screenshot OCR to honor the configured agent streaming transport while preserving image input and OCR metadata parsing.
## Impact
The change affects the screenshot OCR client, the shared LiteLLM Responses message translation, focused tests, and agent configuration documentation. It does not change the local HTTP API or screenshot note format.
@@ -0,0 +1,103 @@
## MODIFIED Requirements
### Requirement: Meeting screenshots are captured into assistant context
Meeting Assistant SHALL expose a configurable screenshot hotkey.
When a meeting is active and the screenshot hotkey is pressed, Meeting Assistant SHALL capture the currently active window.
The screenshot image SHALL be saved into a configurable attachments folder for the assistant context note. By default, the folder SHALL be `Attachments` beside the assistant context note.
After the image is saved, Meeting Assistant SHALL append a markdown image link to the assistant context note with a meeting-relative timestamp that correlates to transcript timestamps.
Meeting Assistant SHALL allow optional screenshot OCR configuration with endpoint URL, API key or key environment variable, model, prompt, and timeout.
When screenshot OCR is configured, Meeting Assistant SHALL send the screenshot and prompt to the configured OpenAI-compatible Responses endpoint and append the model result after the screenshot link in the assistant context note.
Screenshot OCR SHALL honor the configured `MeetingAssistant:Agent:UseStreaming` transport selection.
When streaming is enabled, screenshot OCR SHALL consume the Responses result through the supported OpenAI Responses and Agent Framework Server-Sent Events adapter.
When streaming is disabled, screenshot OCR SHALL consume the result through the supported non-streaming OpenAI Responses client and adapter.
The screenshot OCR prompt SHALL ask the model to return pixel crop coordinates when it can confidently isolate only the presentation, shared screen, or similarly relevant meeting content.
When OCR returns valid crop coordinates within the original image bounds, Meeting Assistant SHALL save a cropped PNG beside the original screenshot and SHALL link the cropped image before the OCR result in the assistant context note.
When OCR returns no crop coordinates or invalid crop coordinates, Meeting Assistant SHALL keep the original screenshot link and OCR result without writing a cropped image.
After transcription finishes and before summarization starts, Meeting Assistant SHALL scan the meeting note for user-authored Obsidian image embeds and Markdown image embeds.
When configured screenshot OCR is enabled and the meeting note contains image embeds, Meeting Assistant SHALL append each resolvable image to the assistant context note, state that the image came from the meeting note, preserve the original embed text for cross-reference, and run OCR for the linked image.
Meeting-note image OCR SHALL NOT copy the image file, SHALL NOT write crop images, SHALL NOT add attendees from OCR metadata, and SHALL NOT modify the meeting note.
Meeting Assistant SHALL wait for meeting-note image OCR to finish or time out before transitioning the assistant context to summarizing.
When screenshot OCR fails or times out, Meeting Assistant SHALL write the failure status into the assistant context note with a retry link for that exact screenshot.
When the screenshot OCR retry link is activated, Meeting Assistant SHALL rerun OCR for the saved screenshot and replace that screenshot's OCR block in the assistant context note.
When screenshot OCR is not configured, Meeting Assistant SHALL skip OCR and keep the screenshot link.
The default OCR prompt SHALL explain that the image is from a meeting and ask the model to identify who is talking, who is presenting, what is presented, capture slide text in markdown, convert diagrams to Mermaid when possible, indicate whether visible people are clearly the exact meeting participants or only a partial result, return crop coordinates only for confidently isolated presentation/shared-screen content, and otherwise describe the scene.
#### Scenario: Screenshot is linked with meeting timestamp
- **GIVEN** a meeting started at `10:00:00`
- **WHEN** the user captures a screenshot at `10:03:05`
- **THEN** Meeting Assistant saves the screenshot under the configured attachments folder
- **AND** appends a markdown image link to assistant context with timestamp `[00:03:05]`
#### Scenario: OCR result is appended after screenshot
- **GIVEN** screenshot OCR is configured
- **WHEN** the user captures a screenshot
- **THEN** Meeting Assistant appends the screenshot link to assistant context
- **AND** appends the OCR result for that screenshot after the link when processing completes
#### Scenario: Streaming screenshot OCR is assembled
- **GIVEN** screenshot OCR is configured
- **AND** `MeetingAssistant:Agent:UseStreaming` is `true`
- **WHEN** the Responses endpoint returns screenshot OCR output as Server-Sent Events
- **THEN** Meeting Assistant sends the prompt and screenshot as one multimodal Responses message
- **AND** appends the assembled OCR text without a JSON document parse failure
#### Scenario: Screenshot OCR streaming can be disabled
- **GIVEN** screenshot OCR is configured
- **AND** `MeetingAssistant:Agent:UseStreaming` is `false`
- **WHEN** Meeting Assistant requests screenshot OCR
- **THEN** it uses the supported non-streaming Responses client and adapter
- **AND** preserves the prompt and screenshot image input
#### Scenario: OCR crop is saved and linked before OCR text
- **GIVEN** screenshot OCR is configured
- **AND** OCR returns valid crop coordinates for a shared screen
- **WHEN** OCR processing completes
- **THEN** Meeting Assistant saves a cropped screenshot beside the original image
- **AND** links the cropped screenshot before the OCR text in assistant context
#### Scenario: Meeting note image embeds are OCRed before summarization
- **GIVEN** screenshot OCR is configured
- **AND** the meeting note contains `![[whiteboard.png]]`
- **AND** the meeting note contains `![Diagram](attachments/diagram.png)`
- **WHEN** transcription finishes
- **THEN** Meeting Assistant appends both images to the assistant context as images from the meeting note
- **AND** includes the original embed text for each image
- **AND** runs OCR for each image without copying files, writing crop images, adding attendees, or modifying the meeting note
- **AND** waits for this OCR to finish or time out before transitioning to summarizing
#### Scenario: OCR failure can be retried for the same screenshot
- **GIVEN** screenshot OCR is configured
- **AND** OCR fails or times out for a captured screenshot
- **WHEN** Meeting Assistant writes the OCR failure status
- **THEN** the assistant context includes a retry link for that exact screenshot
- **WHEN** the retry link is activated
- **THEN** Meeting Assistant reruns OCR against the saved screenshot
- **AND** replaces that screenshot's OCR block with the retry result
#### Scenario: OCR is skipped when not configured
- **GIVEN** screenshot OCR is not configured
- **WHEN** the user captures a screenshot
- **THEN** Meeting Assistant saves and links the screenshot without calling a model endpoint
#### Scenario: OCR reports whether visible people are complete or partial
- **WHEN** Meeting Assistant uses the built-in screenshot OCR prompt
- **THEN** the prompt asks the model to state whether the screenshot clearly shows exactly who is in the meeting or only a partial participant result
@@ -0,0 +1,15 @@
## 1. Streaming screenshot OCR
- [x] 1.1 Add a failing screenshot-client behavior test for an SSE response with prompt and image input.
- [x] 1.2 Route screenshot OCR through the shared Responses client and add multimodal message translation.
## 2. Non-streaming compatibility
- [x] 2.1 Add behavior coverage proving `Agent:UseStreaming=false` preserves non-streaming screenshot OCR and image input.
- [x] 2.2 Document that screenshot OCR inherits the agent streaming setting.
## 3. Verification
- [x] 3.1 Refactor the touched screenshot and shared client paths for DRYness, SOLID boundaries, and simplicity while preserving behavior.
- [x] 3.2 Run focused tests, the full solution tests, and strict OpenSpec validation.
- [x] 3.3 Restart Meeting Assistant only while idle and verify screenshot OCR against the deployed LiteLLM endpoint.
@@ -6,4 +6,4 @@
- [x] Add workflow rule completion/error logging.
- [x] Update workflow engine documentation.
- [x] Run focused tests and `openspec validate resilient-transcript-workflow --strict`.
- [ ] Follow up later: investigate anomalous temporary-recording disk-full `IOException` despite small expected WAV size.
- [x] Follow up later: investigate anomalous temporary-recording disk-full `IOException` despite small expected WAV size. Closed for now because the anomaly has not recurred.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-01
@@ -0,0 +1,30 @@
# Design
## Event Model
Add a workflow event type, `attendee_added`, that carries the attendee string being added. The event is used as a transformation hook before a caller stores the attendee in the meeting note. Rules can match the event with `equals`, `contains`, or `regex` trigger filters over the current attendee value.
## Transformation Contract
The workflow engine exposes `TransformAttendeeAsync`, mirroring `TransformTranscriptLineAsync`. During `attendee_added` events:
- conditions can read `attendee.name`,
- Razor templates can read `Model.Attendee.Name`,
- `set_property attendee.name` mutates the attendee value returned to the caller.
Other workflow steps keep their existing semantics, but the intended transformation path is `set_property attendee.name`.
Rules triggered by `attendee_added` are limited to that transformation path so the value transform does not hide unrelated note or context side effects.
Direct meeting note file edits remain outside the workflow event model. If a user or agent writes a meeting note file/frontmatter directly, Meeting Assistant preserves that content verbatim and does not emit `attendee_added`.
## Call Sites
Callers transform attendee candidates before writing them:
- metadata and accepted prompt metadata after canonicalization,
- workflow `add_attendee` steps before de-duplication,
- screenshot OCR candidates before canonicalization and merge,
- summary-agent `add_attendee` tool calls before duplicate checks and note writes,
- speaker identity attendee additions before adding matched names.
Unchanged metadata attendees preserve the existing canonicalizer output, including email display strings.
@@ -0,0 +1,17 @@
## Why
Attendees enter Meeting Assistant from several paths: calendar metadata, accepted recording prompts, workflow actions, screenshot OCR, speaker identity updates, and summarizer add-attendee tools. Today those paths can normalize display names, but they cannot apply local transformation rules such as trimming company suffixes, replacing aliases, or cleaning attendee strings consistently.
## What Changes
- Add an `attendee_added` workflow trigger that runs whenever Meeting Assistant adds an attendee to a meeting note.
- Allow `attendee_added` triggers to filter the added attendee with full equality, substring, or regex matching.
- Expose `attendee.name` to workflow conditions and Razor templates.
- Allow `set_property` to update `attendee.name` during `attendee_added` events, mirroring transcript-line transformation.
- Apply attendee transformations to attendees added by metadata import, prompted-start metadata, workflow `add_attendee`, screenshot OCR, and summary/workflow editor agent tools.
## Impact
- Workflow engine, rules schema validation, and workflow documentation.
- Attendee write paths in recording metadata, screenshot OCR, summary tools, and workflow actions.
- Behavior tests for workflow transformation and representative attendee-add sources.
@@ -0,0 +1,29 @@
## MODIFIED Requirements
### Requirement: Meeting automation rules support lifecycle triggers
Meeting Assistant SHALL support rule triggers for `created`, `state_transition`, `speaker_identified`, `transcript_line`, and `attendee_added`.
An `attendee_added` trigger MAY filter by `equals`, `contains`, or `regex` against the attendee value being added. `equals` and `contains` filters SHALL match case-insensitively.
Meeting Assistant SHALL apply attendee-added transformations before storing attendees added from meeting metadata, accepted recording-prompt metadata, workflow `add_attendee` steps, screenshot OCR, speaker identity updates, and summarizer add-attendee tools.
#### Scenario: Attendee added rule filters and transforms a metadata attendee
- **GIVEN** a configured rule that triggers on added attendees containing `@contoso.com`
- **AND** the rule sets `attendee.name` to a display name derived from the added attendee
- **WHEN** Meeting Assistant adds attendee `Ada Lovelace <ada@contoso.com>` from meeting metadata
- **THEN** the stored meeting attendee is the transformed attendee name
- **AND** the original attendee string is not stored
### Requirement: Meeting automation rules support conditions and steps
Meeting Assistant SHALL expose the added attendee name to `attendee_added` rule conditions and Razor step templates as `attendee.name`.
The `set_property` step SHALL support setting `attendee.name` during `attendee_added` events.
Rules with an `attendee_added` trigger SHALL be limited to transforming `attendee.name` with `set_property`.
Direct meeting note file writes SHALL NOT emit `attendee_added` or transform attendee values.
#### Scenario: Attendee added rules can use regex and Razor
- **GIVEN** a configured `attendee_added` rule with a regex filter over the added attendee
- **WHEN** Meeting Assistant adds an attendee matching the regex
- **THEN** the rule can set `attendee.name` with a Razor template using `Model.Attendee.Name`
@@ -0,0 +1,8 @@
# Tasks
- [x] Add OpenSpec scenarios for attendee-added trigger filtering and transformation.
- [x] Add a failing workflow-engine behavior test for transforming an added attendee through `attendee_added`.
- [x] Implement attendee-added workflow event, trigger filters, `attendee.name` condition/template value, and `set_property attendee.name`.
- [x] Apply attendee transformations from metadata/prompted starts, workflow `add_attendee`, screenshot OCR, and agent attendee tools.
- [x] Update workflow engine documentation.
- [x] Run focused tests plus `openspec validate transform-added-attendees --strict`.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-17
@@ -0,0 +1,45 @@
## Context
Meeting Assistant already creates one assistant context artifact per meeting and gives both the automatic summary agent and the interactive agent window tools that can read and append to it. The current summary prompt calls the file a notebook, while the interactive agent prompt describes only generic artifact repair. Neither prompt establishes assistant context as the durable handoff point for uncertainty, failures, assumptions, repairs, and conclusions about one meeting.
## Goals / Non-Goals
**Goals:**
- Make the assistant context's meeting-specific memory role explicit to both agents.
- Preserve summarization problems, missing information, and assumptions for later investigation.
- Make the interactive agent consult that memory before repairing a meeting or summary.
- Make the interactive agent record completed fixes and conclusions for later agents.
**Non-Goals:**
- Add a new artifact type, schema, section format, or database.
- Automatically rewrite or summarize existing assistant context files.
- Change tool permissions or allow agents to access artifacts outside existing configured scopes.
- Override an explicitly configured custom summary-agent prompt with built-in summary guidance.
## Decisions
### Extend the existing instruction builders
The automatic summary behavior will be added to `MeetingSummaryInstructionBuilder.DefaultInitialPrompt`. This preserves the existing contract that a configured `Agent:InitialPrompt` replaces the built-in summary prompt.
The interactive behavior will be included in the instruction builder's always-appended meeting-artifact guidance. This keeps the meeting-memory reminder available even when the interactive agent has a custom initial prompt, alongside the tool capabilities that Meeting Assistant already appends.
Alternative considered: implement automatic interception or mandatory writes whenever an agent encounters uncertainty. The application cannot reliably infer those semantic events from arbitrary model turns, so explicit instructions are the smallest dependable mechanism.
### Keep memory entries append-oriented and concise
Agents will be told to append problems, missing information, assumptions, fixes, and conclusions to the matching assistant context. Existing `write_context` behavior already appends by default, which preserves earlier observations and avoids introducing a structured migration.
Alternative considered: define mandatory headings or a machine-readable memory schema. That would add formatting and compatibility obligations without being necessary for the requested agent handoff behavior.
### Keep user-facing summary content separate
Assistant context remains internal meeting memory; the generated summary remains the user-facing artifact. Assumptions and missing evidence can inform explicit uncertainty in the summary, but diagnostic notes and repair history belong in assistant context.
## Risks / Trade-offs
- [Repeated agents may append duplicate observations] → Instruct agents to read existing context first and keep entries concise.
- [Assistant context can grow over time] → Preserve existing ranged and tail reads; no new retention mechanism is introduced.
- [Configured custom summary prompts omit the new built-in guidance] → Preserve the documented replacement semantics and leave responsibility with the custom prompt author.
@@ -0,0 +1,27 @@
## Why
The automatic summarizer and interactive agent window can encounter uncertainty, missing information, and repair conclusions that are useful to later work on the same meeting. Those observations currently depend on the active agent turn instead of being consistently preserved in the meeting's assistant context.
## What Changes
- Treat each assistant context file as meeting-specific memory shared by agents working on that meeting.
- Guide the automatic summarizer to record unexpected problems, missing information, and assumptions in assistant context.
- Guide the interactive agent window to consult assistant context when repairing a meeting or summary.
- Guide the interactive agent window to append its fixes and conclusions to assistant context for future work on that meeting.
## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `meeting-summary`: Define the automatic summarizer's responsibility to persist uncertainty and problems in meeting-specific assistant context.
- `meeting-session`: Define how the interactive agent window uses and updates assistant context during meeting and summary repairs.
## Impact
- Summary-agent default instructions and their behavior tests.
- Interactive agent-window instructions and their behavior tests.
- Agent documentation describing assistant context usage.
@@ -0,0 +1,14 @@
## ADDED Requirements
### Requirement: Interactive agent uses assistant context as meeting memory
The interactive agent window instructions SHALL identify each assistant context file as meeting-specific memory.
When the user asks the interactive agent to fix or investigate a meeting or summary, the instructions SHALL direct the agent to read the matching assistant context for clues about problems, missing information, assumptions, prior fixes, and conclusions.
After repairing a meeting or summary, the instructions SHALL direct the agent to append a concise record of its fixes and conclusions to the matching assistant context.
#### Scenario: Interactive agent repairs a meeting artifact
- **GIVEN** a meeting has an assistant context file
- **WHEN** the user asks the interactive agent to fix that meeting or its summary
- **THEN** the agent instructions direct it to inspect the matching assistant context for relevant meeting-specific memory
- **AND** direct it to append its fixes and conclusions to that assistant context
@@ -0,0 +1,12 @@
## ADDED Requirements
### Requirement: Summary agent preserves meeting-specific working memory
When Meeting Assistant uses the built-in summary-agent instructions, those instructions SHALL identify assistant context as persistent meeting-specific memory.
The built-in instructions SHALL direct the summary agent to append unexpected problems, missing information, and assumptions encountered while summarizing to assistant context.
#### Scenario: Summarizer encounters uncertainty
- **GIVEN** the built-in summary-agent instructions are in use
- **WHEN** the summary agent encounters an unexpected problem, cannot find needed information, or must make an assumption
- **THEN** its instructions direct it to append a concise record to the meeting's assistant context
- **AND** later agents can discover that record when working on the same meeting
@@ -0,0 +1,14 @@
## 1. Automatic Summary Agent
- [x] 1.1 Add a behavior test that requires the built-in summary instructions to treat assistant context as meeting-specific memory and record problems, missing information, and assumptions.
- [x] 1.2 Update the built-in summary-agent guidance to satisfy the memory behavior.
## 2. Interactive Agent Window
- [x] 2.1 Add a behavior test that requires the interactive instructions to inspect assistant context during meeting or summary repair and record fixes and conclusions.
- [x] 2.2 Update the interactive agent-window guidance to satisfy the memory behavior.
## 3. Documentation and Validation
- [x] 3.1 Document assistant context as shared meeting-specific agent memory.
- [x] 3.2 Run focused behavior tests, the full test suite, and strict OpenSpec validation.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-27
@@ -0,0 +1,40 @@
## Context
`LiteLlmResponsesChatClient` exposes a non-streaming `IChatClient` interface and currently buffers every successful `/v1/responses` body before parsing it as one JSON document. The deployed LiteLLM `chatgpt/gpt-5.5` route instead returns Responses Server-Sent Events. Microsoft.Extensions.AI.OpenAI already provides a Responses streaming adapter that maps the OpenAI SDK's typed SSE updates into `ChatResponseUpdate` values, including text, reasoning, function calls, response metadata, and usage.
That adapter uses `FunctionCallContent.CreateFromParsedArguments`, which records argument-mapping failures on the function call instead of throwing them from the response parser. The default function invoker does not itself prevent invocation when that property is populated, so Meeting Assistant must turn the recorded parse failure into a matching function result before calling the tool.
## Goals / Non-Goals
**Goals:**
- Preserve the existing non-streaming `IChatClient` contract while selecting the supported Responses streaming or non-streaming path through configuration.
- Delegate SSE framing, event deserialization, streamed function-call assembly, response metadata, and usage mapping to the OpenAI SDK and Microsoft.Extensions.AI.OpenAI adapter.
- Preserve the original encoded function arguments when forwarding conversation history.
- Return a safe, structured invalid-arguments result for malformed argument JSON without invoking the requested tool.
- Apply the guarded invocation behavior to both summary and workflow-editor agent pipelines that use this Responses client.
**Non-Goals:**
- Expose token-by-token upstream streaming to the UI.
- Replace Microsoft.Extensions.AI function invocation or implement general JSON Schema validation.
- Recover from an incomplete or malformed Responses event stream that has no usable completed output.
## Decisions
1. Add `Agent:UseStreaming`, defaulting to `true`, and send summary requests through the matching OpenAI SDK method. The streaming path uses `ResponsesClient.CreateResponseStreamingAsync` and collects the Microsoft.Extensions.AI.OpenAI typed updates into one `ChatResponse`; the non-streaming path uses `ResponsesClient.CreateResponseAsync` and its supported `AsChatResponse` adapter.
2. Keep the existing JSON payload builder because it owns Meeting Assistant-specific compaction input, tool serialization, reasoning settings, retry diagnostics, and initiator behavior. Deserialize that payload into the OpenAI SDK's `CreateResponseOptions`, set its streaming flag from configuration, and let the selected SDK method own the HTTP response protocol.
The SDK deserializes `ResponseItem` values by their `type` discriminator. Message items must therefore include `"type": "message"` before the JSON payload is converted to `CreateResponseOptions`; otherwise the SDK reserializes them as `"type": "unknown"` and the provider rejects the request.
3. Rely on the framework adapter's `FunctionCallContent.CreateFromParsedArguments` mapping for function-call argument decoding and raw response preservation. A shared function-invocation callback checks the recorded parse exception before invocation. For invalid JSON it returns a structured `invalid_tool_arguments` value that the framework associates with the original call ID; otherwise it delegates to the actual function.
4. Configure both agent pipelines with the shared guarded invoker. This avoids duplicating the safety decision and prevents zero-argument tools from accidentally running when malformed JSON would otherwise map to an empty argument set.
## Risks / Trade-offs
- **[Provider event variants]** A provider could emit events outside the OpenAI Responses schema. → Use the maintained OpenAI SDK parser and fail diagnostically for genuinely incompatible streams instead of maintaining local event variants.
- **[Buffered result]** The summary pipeline does not expose token-by-token updates to its caller. → Collect the framework updates only at the existing non-streaming boundary; the HTTP response itself remains streamed and incrementally parsed.
- **[Experimental API]** The OpenAI Responses adapter is marked experimental in the currently referenced package. → Keep it behind `LiteLlmResponsesChatClient`, which isolates future package API changes from the rest of Meeting Assistant.
- **[Error disclosure]** Raw parser exceptions may contain implementation details. → Return a stable error code and concise validation message rather than exception text or the malformed arguments.
@@ -0,0 +1,26 @@
## Why
The configured LiteLLM Responses endpoint can return successful responses as Server-Sent Events even when the request sets `stream: false`. Meeting Assistant currently treats every successful body as one JSON document, and it also lets malformed function-call argument JSON escape as a fatal parse exception, so either condition can abort an otherwise recoverable summary run.
## What Changes
- Consume the configured LiteLLM Responses endpoint through the supported OpenAI Responses streaming transport and Agent Framework adapter.
- Assemble completed response output items and usage metadata from Responses SSE events before returning them through the existing non-streaming chat-client interface.
- Add an agent setting that selects streaming or non-streaming Responses transport, with streaming enabled by default.
- Preserve valid Responses item discriminators when translating agent messages through the OpenAI SDK request model.
- Treat malformed function-call arguments as invalid tool input that is returned to the agent for correction without invoking the tool or terminating the summary run.
- Add behavior tests for streamed text, streamed function calls, and invalid function-call arguments.
## Capabilities
### New Capabilities
- None.
### Modified Capabilities
- `meeting-summary`: Make the existing Responses-based summary pipeline interoperable with SSE responses and resilient to malformed tool-call input.
## Impact
The change affects agent configuration, the custom LiteLLM Responses chat client, its summary-agent integration, and focused tests. It does not change the public HTTP API.
@@ -0,0 +1,36 @@
## ADDED Requirements
### Requirement: Summary agents tolerate Responses event streams and malformed tool arguments
Meeting Assistant SHALL consume successful summary-agent Responses results through the supported OpenAI Responses and Agent Framework Server-Sent Events adapter.
When a Responses event stream delivers completed output items separately from the final response metadata, Meeting Assistant SHALL assemble those output items into one agent response while preserving final response metadata and usage.
Meeting Assistant SHALL provide an agent setting that selects streaming or non-streaming Responses transport. Streaming SHALL be enabled by default. When streaming is disabled, Meeting Assistant SHALL use the supported non-streaming OpenAI Responses client and adapter.
When a returned function call contains arguments that are not a valid JSON object, Meeting Assistant SHALL NOT invoke the requested function and SHALL return an invalid-tool-arguments result associated with the original call ID to the agent so it can correct the call.
When Meeting Assistant translates chat messages into a Responses request, every message input item SHALL retain the `message` item discriminator required by the OpenAI SDK and Responses API.
#### Scenario: Streamed text response is assembled
- **WHEN** the configured Responses endpoint returns completed message output in Server-Sent Events followed by final response metadata
- **THEN** the summary agent receives the completed message text, response metadata, and usage without a JSON document parse failure
#### Scenario: Streamed function call is assembled
- **WHEN** the configured Responses endpoint returns a completed function-call output item in Server-Sent Events
- **THEN** the summary agent receives the function call with its call ID, function name, and parsed arguments
#### Scenario: Streaming transport can be disabled
- **WHEN** `MeetingAssistant:Agent:UseStreaming` is `false`
- **THEN** the summary agent requests a non-streaming Responses result
- **AND** converts the response through the supported OpenAI Responses adapter
#### Scenario: Chat message input remains a Responses message
- **WHEN** the summary agent sends a user or assistant chat message through the Responses client
- **THEN** the outbound Responses input item has type `message`
- **AND** the request does not contain an `unknown` input item type
#### Scenario: Malformed function arguments are returned to the agent
- **WHEN** the model returns a function call whose arguments are not a valid JSON object
- **THEN** Meeting Assistant does not invoke the requested function
- **AND** sends a function result with an invalid-tool-arguments error for the original call ID back to the agent
- **AND** allows the agent loop to continue
@@ -0,0 +1,24 @@
## 1. Responses SSE compatibility
- [x] 1.1 Add a failing client behavior test for a streamed text response with final metadata and usage.
- [x] 1.2 Add failing configuration and client behavior tests for selecting non-streaming Responses transport.
- [x] 1.3 Route Responses requests through the Agent Framework/OpenAI SDK streaming or non-streaming adapter according to configuration.
- [x] 1.4 Add behavior coverage for streamed function-call output.
## 2. Invalid tool-argument recovery
- [x] 2.1 Add a failing agent-loop behavior test proving malformed function-call JSON does not invoke the tool and is returned to the agent.
- [x] 2.2 Preserve argument parse failures on function-call content and add the shared guarded function invoker.
- [x] 2.3 Apply guarded function invocation to the summary and workflow-editor agent pipelines.
## 3. Verification
- [x] 3.1 Refactor the touched response and invocation paths for DRYness, SOLID boundaries, and simplicity while preserving behavior.
- [x] 3.2 Run focused tests, the full solution tests, and strict OpenSpec validation.
- [x] 3.3 Verify the client behavior against the deployed LiteLLM Responses endpoint and restart Meeting Assistant only after confirming it is idle.
## 4. Responses message-item compatibility
- [x] 4.1 Add a failing client behavior test proving an outbound chat message retains the `message` discriminator through SDK serialization.
- [x] 4.2 Emit the Responses `message` discriminator for chat message input items.
- [x] 4.3 Run focused and full tests, validate OpenSpec strictly, restart while idle, and retry the failed summary through the local API.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-05
@@ -0,0 +1,69 @@
## Context
Standalone Outlook enrichment already selects a single suitable Teams appointment and already contains a fixed five-minute upcoming-start window, but that behavior is not represented in the accepted specification. The selector intentionally returns no metadata when the suitable candidates are ambiguous. Calendar notifications avoid that ambiguity because each notification carries one cached appointment and its metadata.
The prompt scheduler currently offers `Record` and `Skip`. Choosing `Record` while a meeting is active stops that meeting and starts a new run with the prompted metadata. Metadata application is currently private to recording startup/background lookup, so the scheduler has no safe way to enrich the active run in place.
## Goals / Non-Goals
**Goals:**
- Specify and preserve the existing five-minute pre-start Outlook metadata grace period.
- Preserve conservative standalone selection when more than one appointment is suitable.
- Offer an appointment-specific metadata action only when a recording is active and the prompt carries metadata.
- Apply that metadata atomically to the active run without interrupting capture or transcription.
- Prevent a slower standalone Outlook lookup from overwriting explicitly attached prompt metadata.
- Reuse existing attendee canonicalization, attendee-added workflow transformations, attendee import limits, and artifact rendering.
**Non-Goals:**
- Automatically choose among concurrent appointments.
- Change the existing `Yes` behavior that finishes the current recording and starts the prompted appointment as a new recording.
- Add a new workflow trigger or replay an already-completed lifecycle state transition.
- Guarantee an exact action-button row layout that the native Windows toast renderer does not expose to the application.
## Decisions
### Keep the five-minute grace period in the shared candidate selector
The current selector already treats exactly one appointment starting within five minutes as eligible when there is no suitable overlap. The change will codify this behavior in OpenSpec and retain its behavior test. Candidate selection remains conservative: multiple suitable overlaps or multiple upcoming candidates return no selection.
This keeps manual enrichment independent of the calendar prompt cache. Using the prompt cache for all metadata lookup was considered, but it would couple ordinary recording startup to an optional hosted feature and its sync freshness.
### Put active-recording capability on the prompt request
The scheduler will snapshot whether the prompt can attach metadata when it calls the prompt service. The Windows adapter will add a third `Add metadata to current meeting` background action after the existing `Yes` and `No` actions only when that flag is true. The response enum will carry a distinct attach result, so the callback still identifies the exact cached appointment.
The native toast API controls final action layout and does not provide a reliable per-button full-row placement contract. Adding the action after the two short actions gives the renderer the best available ordering while keeping the label explicit.
### Add one coordinator operation for explicit active-run metadata
`IMeetingPromptRecordingController` will expose an attach operation backed by `MeetingRecordingCoordinator`. Under the coordinator gate, it will require a currently capturing run, mark that run as explicitly assigned, re-read the latest meeting note, apply the shared metadata rules, save the note, and refresh assistant-context and transcript metadata. Capture and transcription continue unchanged.
Centralizing the mutation in the coordinator keeps run ownership, artifact paths, profile options, and serialization under the same synchronization boundary. Direct file mutation from the scheduler was rejected because it could race recording lifecycle writes and would bypass attendee normalization and workflow transformations.
### Explicit prompt metadata wins over background lookup
Each recording run will track whether appointment metadata was explicitly assigned. The background Outlook task will check this flag before and after acquiring the coordinator gate. If explicit prompt metadata has already been attached, the background result is discarded. If the background update wins the gate first, the later explicit action overwrites it, so the user's appointment choice remains authoritative.
### Do not replay lifecycle transitions
Attendee-added transformations run as part of the shared metadata application path. The meeting has already transitioned from `collecting metadata` to `transcribing`, so the attach action will not fabricate or replay that state transition. Existing live speaker matching already observes changes to meeting-note attendees.
## Risks / Trade-offs
[Native toast may not render the third action as a full-width row] -> Keep it as the final, clearly labelled action and let Windows choose the physical layout.
[The recording stops before the user activates the notification] -> Recheck active capture under the coordinator gate and leave artifacts unchanged if there is no current recording.
[Metadata attachment replaces title and attendee metadata] -> Reuse the established prompted-start semantics so the selected appointment becomes authoritative while preserving the note body and other user-authored content.
[A delayed background lookup races the explicit action] -> Record explicit assignment on the run and make the explicit appointment win regardless of completion order.
## Migration Plan
No data or configuration migration is required. The notification gains one conditional action, and existing `Yes`/`No` activation arguments remain valid. Rollback removes the action and coordinator method without changing stored meeting artifacts.
## Open Questions
None.
@@ -0,0 +1,28 @@
## Why
Manual recordings started shortly before an appointment currently miss Outlook metadata, while concurrent appointments make automatic lookup ambiguous. Once an appointment-specific recording notification appears during an active recording, the user also has no way to attach that exact appointment's metadata without stopping the current meeting.
## What Changes
- Treat a Teams appointment that starts within five minutes after a manual recording begins as current for Outlook metadata enrichment.
- Keep standalone metadata lookup conservative when multiple in-progress appointments match, or when multiple appointments fall in the grace window and none is already in progress.
- When a calendar recording prompt is shown during an active recording, add an `Add metadata to current meeting` action below the existing affirmative and negative actions when the Windows notification layout permits it.
- Apply the prompted appointment's title, eligible attendees, agenda, and scheduled end to the active meeting without stopping or starting a recording.
- Reuse the normal attendee transformation and artifact-update behavior when prompted metadata is attached to an active meeting.
## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `meeting-session`: Extend Outlook enrichment with a five-minute pre-start grace period and allow appointment-specific prompt metadata to be attached to an active meeting.
## Impact
- Outlook current-meeting candidate selection and its Windows provider.
- Calendar prompt scheduling, response contracts, and Windows toast layout/activation handling.
- The active recording controller/coordinator metadata update surface.
- Meeting-note, assistant-context, transcript metadata, and workflow behavior tests.
@@ -0,0 +1,184 @@
## MODIFIED Requirements
### Requirement: Windows Outlook enrichment is optional
Meeting Assistant SHALL gate Outlook Classic COM enrichment behind the Windows compilation target.
For standalone metadata lookup, Meeting Assistant SHALL first consider Teams appointments that are in progress with at least five minutes remaining. If exactly one such appointment exists, Meeting Assistant SHALL select it even when another appointment starts within five minutes.
If no suitable in-progress appointment exists, Meeting Assistant SHALL consider Teams appointments that start within five minutes after recording starts and SHALL select one only when exactly one such upcoming appointment exists.
When the Windows build starts a meeting and Outlook Classic yields one appointment through that selection order, Meeting Assistant SHALL copy the appointment title to the meeting note and copy the appointment agenda and scheduled end time to the assistant-context frontmatter.
When more than one appointment exists in the applicable in-progress or upcoming selection group, Meeting Assistant SHALL leave standalone metadata unselected.
Meeting Assistant SHALL exclude canceled Outlook appointments from current Teams appointment metadata lookup.
Meeting Assistant SHALL copy the appointment attendees to the meeting note only when the raw appointment attendee count is less than or equal to the configured `Recording:MaxMetadataAttendeeImportCount`. The default maximum SHALL be 30 attendees.
The agenda SHALL be extracted from the appointment body content before the Teams join separator or Teams join text.
#### Scenario: Current Teams appointment enriches meeting artifacts
- **WHEN** a Windows build starts a meeting while Outlook Classic selects exactly one Teams appointment through the standalone selection order
- **THEN** Meeting Assistant uses the appointment subject as the meeting title
- **AND** writes the appointment attendees into meeting note frontmatter when the raw attendee count is within the configured import limit
- **AND** writes the appointment agenda into assistant-context frontmatter
- **AND** writes the appointment end time as `scheduled_end` into assistant-context frontmatter
#### Scenario: Meeting starting within five minutes enriches an early recording
- **GIVEN** no suitable Teams appointment is already in progress
- **AND** exactly one Teams appointment starts one minute after the recording start time
- **WHEN** a Windows build starts the meeting recording
- **THEN** Meeting Assistant selects that upcoming appointment's metadata
#### Scenario: In-progress meeting takes priority over an upcoming meeting
- **GIVEN** exactly one Teams appointment is in progress with at least five minutes remaining
- **AND** another Teams appointment starts within five minutes after the recording start time
- **WHEN** a Windows build starts the meeting recording
- **THEN** Meeting Assistant selects the in-progress appointment's metadata
#### Scenario: Canceled appointment is ignored during metadata lookup
- **GIVEN** Outlook Classic exposes one canceled suitable Teams appointment
- **AND** Outlook Classic exposes one active suitable Teams appointment at the same time
- **WHEN** a Windows build starts a meeting
- **THEN** Meeting Assistant selects the active appointment metadata
#### Scenario: Oversized attendee list is not imported
- **GIVEN** the configured metadata attendee import limit is 30
- **WHEN** a Windows build starts a meeting while Outlook Classic exposes exactly one suitable Teams appointment with 31 attendees
- **THEN** Meeting Assistant uses the appointment subject as the meeting title
- **AND** does not write the appointment attendees into meeting note frontmatter
- **AND** writes the appointment agenda into assistant-context frontmatter
- **AND** writes the appointment end time as `scheduled_end` into assistant-context frontmatter
#### Scenario: Outlook is unavailable or metadata lookup is ambiguous
- **WHEN** Outlook Classic is unavailable or more than one Teams appointment exists in the applicable in-progress or upcoming selection group
- **THEN** Meeting Assistant starts the recording with the default generated meeting title and empty agenda
- **AND** omits `scheduled_end` from assistant-context frontmatter
### Requirement: Outlook Teams meetings can prompt recording start
Meeting Assistant SHALL enable scheduled Outlook Classic calendar checks for recording-start prompts by default.
When scheduled recording prompts are enabled on Windows, Meeting Assistant SHALL periodically read the user's Outlook Classic calendar appointments for the current local day through COM into an in-memory cache.
Meeting Assistant SHALL default the Outlook calendar sync interval to 30 minutes when scheduled recording prompts are enabled.
Meeting Assistant SHALL schedule recording-start prompts from the cached calendar appointments rather than querying Outlook for each prompt.
Meeting Assistant SHALL consider Teams appointments from Outlook calendar data as initial prompt candidates. The detection MAY be extended later for other meeting providers.
Meeting Assistant SHALL exclude canceled Outlook appointments from recording-start prompt candidates.
When a Teams appointment reaches its scheduled start window, Meeting Assistant SHALL show a native Windows app notification asking whether to record the meeting, with affirmative and negative actions.
When a recording is active and the cached appointment has metadata, the recording-start notification SHALL additionally offer an `Add metadata to current meeting` action after the affirmative and negative actions. The native Windows notification renderer MAY choose the physical button layout.
On Windows, the recording-start notification SHALL request reminder-style toast behavior and remain actionable for 5 minutes.
Meeting Assistant SHALL prompt at most once per calendar appointment during a local day, regardless of whether the user accepts, declines, ignores, or attaches metadata from the notification.
If the user accepts the recording prompt while no recording is active, Meeting Assistant SHALL start a new recording normally.
If the user accepts the recording prompt while another recording is active, Meeting Assistant SHALL stop the active recording normally and then start the prompted meeting recording.
When stopping an active recording for an accepted prompt, Meeting Assistant SHALL use the normal stop path so empty or too-short recordings are removed according to existing settings and other completed recordings continue normal transcription, speaker recognition, and summary processing.
When the user accepts a recording prompt, Meeting Assistant SHALL start the recording with the accepted cached appointment's metadata and SHALL NOT perform a separate current-meeting metadata lookup for that prompted start.
Prompted-start metadata SHALL include the accepted appointment title, attendees, agenda, and scheduled end when those values are available from the cached appointment.
Prompted starts SHALL run the normal meeting workflow `created` rules before applying the accepted appointment metadata.
After `created` rules run, prompted starts SHALL apply the accepted appointment metadata and then run the normal `collecting metadata` to `transcribing` state-transition workflow rules with that metadata available in the meeting note.
If the user chooses `Add metadata to current meeting` while that recording is still active, Meeting Assistant SHALL apply the exact cached appointment's title, eligible attendees, agenda, and scheduled end to the active meeting without stopping or starting recording.
Attaching prompted metadata SHALL use the normal attendee canonicalization, attendee import limit, and `attendee_added` workflow transformations, SHALL preserve the meeting note body, and SHALL refresh meeting metadata in the meeting note, assistant context, and transcript artifacts.
Explicitly attached prompted metadata SHALL take precedence over any standalone Outlook metadata lookup still running for that recording.
If no recording is active when the attach action is handled, Meeting Assistant SHALL leave meeting artifacts and recording state unchanged.
#### Scenario: Teams meeting start prompts the user
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** Meeting Assistant has synced Outlook Classic Teams appointments for today
- **WHEN** the appointment reaches its scheduled start window
- **THEN** Meeting Assistant shows a native Windows app notification asking whether to record the meeting
- **AND** the notification remains actionable for 5 minutes
- **AND** marks that appointment as prompted for the day
#### Scenario: Active recording prompt offers metadata attachment
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** a meeting recording is active
- **AND** the due cached appointment has metadata
- **WHEN** Meeting Assistant shows the appointment's recording-start notification
- **THEN** the notification includes `Yes`, `No`, and `Add metadata to current meeting` actions
- **AND** orders the metadata action after `Yes` and `No`
#### Scenario: Idle prompt omits metadata attachment
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** no meeting recording is active
- **WHEN** Meeting Assistant shows an appointment's recording-start notification
- **THEN** the notification includes the existing affirmative and negative actions
- **AND** does not include `Add metadata to current meeting`
#### Scenario: Canceled Teams meeting does not prompt recording
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** Meeting Assistant has synced a canceled Outlook Classic Teams appointment for today
- **WHEN** the canceled appointment reaches its scheduled start window
- **THEN** Meeting Assistant does not show a recording prompt for that appointment
#### Scenario: Back-to-back cached Teams meetings prompt without another Outlook sync
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** Meeting Assistant has synced two Teams appointments for today that start ten minutes apart
- **WHEN** each appointment reaches its scheduled start window
- **THEN** Meeting Assistant shows a recording prompt for each appointment
- **AND** does not require another Outlook calendar sync between the prompts
#### Scenario: User accepts prompt while idle
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** no meeting recording is active
- **WHEN** the user accepts a Teams meeting recording prompt
- **THEN** Meeting Assistant starts recording normally
#### Scenario: User accepts prompt while already recording
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** a meeting recording is active
- **WHEN** the user accepts a Teams meeting recording prompt
- **THEN** Meeting Assistant stops the active recording normally
- **AND** starts a new recording normally after the stop request
#### Scenario: Accepted prompt supplies meeting metadata
- **GIVEN** scheduled Outlook recording prompts are enabled
- **AND** Meeting Assistant has cached two current Teams appointments with different metadata
- **WHEN** the user accepts the recording prompt for one appointment
- **THEN** Meeting Assistant starts the recording with the accepted appointment's metadata
- **AND** does not perform a standalone current-meeting metadata lookup for that recording
- **AND** runs `created` workflow rules before writing the cached appointment metadata
- **AND** runs `collecting metadata` to `transcribing` workflow rules after writing the cached appointment metadata
#### Scenario: Prompted metadata is attached to the active meeting
- **GIVEN** a meeting recording is active without calendar metadata
- **AND** two concurrent appointment notifications carry different cached metadata
- **WHEN** the user chooses `Add metadata to current meeting` on one notification
- **THEN** Meeting Assistant applies only that notification's appointment metadata to the active meeting
- **AND** keeps the current recording and transcription running
- **AND** does not start a new recording
#### Scenario: Explicit attachment wins over delayed lookup
- **GIVEN** standalone Outlook metadata lookup is still running for an active recording
- **WHEN** the user attaches metadata from an appointment notification
- **AND** the standalone lookup completes later with different metadata
- **THEN** the prompted appointment metadata remains on the active meeting artifacts
#### Scenario: Attach action becomes stale
- **GIVEN** a recording-start notification offered `Add metadata to current meeting`
- **AND** the active recording stops before the action is handled
- **WHEN** the user activates the metadata action
- **THEN** Meeting Assistant does not change completed meeting artifacts
- **AND** does not start a recording
#### Scenario: Prompt is disabled
- **GIVEN** scheduled Outlook recording prompts are disabled
- **WHEN** a Teams appointment reaches its scheduled start
- **THEN** Meeting Assistant does not query Outlook for recording prompt candidates
- **AND** does not show a recording prompt
@@ -0,0 +1,24 @@
## 1. Calendar Selection Contract
- [x] 1.1 Verify the standalone Outlook selector covers a meeting started one minute early and remains ambiguous for concurrent upcoming meetings.
## 2. Prompt Attach Action
- [x] 2.1 Add a failing scheduler behavior test that an active-recording prompt exposes metadata attachment and routes the exact appointment metadata without stop/start.
- [x] 2.2 Implement the prompt request/response and recording-controller attach contract to pass the scheduler behavior test.
- [x] 2.3 Add the conditional `Add metadata to current meeting` Windows toast action and activation mapping while preserving Yes/No behavior.
## 3. Active Meeting Metadata
- [x] 3.1 Add a failing coordinator behavior test for attaching title, eligible transformed attendees, agenda, and scheduled end to active meeting, assistant-context, and transcript artifacts without interrupting recording.
- [x] 3.2 Implement synchronized active-run metadata attachment through the shared metadata application path.
- [x] 3.3 Add a failing race behavior test proving explicit prompted metadata wins over a delayed standalone lookup, then implement the run-level explicit-assignment guard.
- [x] 3.4 Add a stale-action behavior test proving attachment is a no-op after capture stops.
- [x] 3.5 Add a failing attachment behavior test proving a persistence failure does not suppress the fallback Outlook lookup, then mark metadata explicit only after successful persistence.
## 4. Verification
- [x] 4.1 Run focused calendar and recording coordinator tests, then the full test suite and Windows-target build.
- [x] 4.2 Run `openspec validate attach-calendar-metadata-to-active-meeting --strict` and verify the change task list is complete.
- [x] 4.3 Perform the required DRY, SOLID, and KISS refactoring passes with tests after any changes.
- [x] 4.4 Verify the notification and active-recording behavior through the safest available operational surface without interrupting live meeting work.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-04
@@ -0,0 +1,45 @@
## Context
The tray-menu builder currently returns a flat list of semantic actions, while the Windows renderer infers separators from item indexes and the Exit action. During an active recording, the normal stop action is added after the microphone submenu and uses a long implementation-oriented label. This makes the primary meeting-completion action look equivalent to cancel, profile switching, and device selection.
## Goals / Non-Goals
**Goals:**
- Give normal meeting completion the concise label `Finish meeting`.
- Make that action the only item in the section immediately below `Open agent` while recording.
- Keep fine-grained recording controls in a distinct following section.
- Make section boundaries observable in platform-independent menu behavior tests.
**Non-Goals:**
- Change what normal stop, abort, profile switching, or microphone selection does.
- Change idle-menu actions, hotkeys, endpoints, or recording state transitions.
- Add icons, confirmation prompts, or nested submenus.
## Decisions
### Represent section starts in the menu model
Add a section-start flag to `MeetingTaskbarMenuItem`. The Windows renderer will insert a separator before items carrying the flag instead of deriving layout from array indexes and action types.
This keeps layout intent in the platform-independent builder where behavior tests can observe it. Keeping another renderer-only special case was rejected because it would leave the requested prominence untestable without Windows UI automation.
### Build prioritized and fine-grained controls as separate groups
While recording, the builder will add `Open agent`, then `Finish meeting` as a new section, then collect microphone, cancel/discard, and profile-switch actions into a fine-grained group whose first item starts another section. Exit remains the final section.
The action continues to use the existing normal-stop command so transcription, speaker processing, OCR, and summarization semantics do not change.
## Risks / Trade-offs
- **A section flag could produce adjacent separators if assigned carelessly** → The builder marks only the first item of each non-empty group, and the renderer follows those explicit starts.
- **Menu ordering changes while recording** → Limit reordering to the active-recording state; idle and processing actions retain their existing relative order.
## Migration Plan
No configuration or data migration is required. Deploying the updated executable changes only tray-menu presentation. Rollback restores the previous label and grouping.
## Open Questions
None.
@@ -0,0 +1,25 @@
## Why
The active-recording tray menu labels its most important completion action as the verbose `Stop meeting recording and transcribe` and groups it with rarely used controls. Finishing a meeting should be immediately recognizable and visually prioritized during normal use.
## What Changes
- Rename the active-recording stop action to `Finish meeting` without changing its normal stop, transcription, or summary behavior.
- Place `Finish meeting` by itself in the section immediately below `Open agent`.
- Place microphone selection, cancel/discard, and profile-switch controls in a separate lower-priority section.
- Represent tray-menu section boundaries explicitly so ordering and prominence are behavior-testable.
## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `meeting-recording`: Prioritize the normal meeting completion action in the Windows tray menu with a concise label and dedicated section.
## Impact
- Affects the platform-independent tray-menu model/builder, Windows tray-menu rendering, and taskbar behavior tests.
- Does not change recording lifecycle semantics, hotkeys, endpoints, or generated meeting artifacts.
@@ -0,0 +1,62 @@
## MODIFIED Requirements
### Requirement: Windows taskbar icon controls recording
Meeting Assistant SHALL show a Windows taskbar notification icon when running on Windows.
The taskbar icon SHALL indicate whether the newest meeting process is idle, actively recording, or post-recording processing/summarizing.
When a new meeting is actively recording while an older stopped meeting is still transcribing, recognizing speakers, or summarizing, the taskbar icon SHALL show the new active recording state.
The taskbar icon right-click menu SHALL expose recording controls based on the current state and configured launch profiles.
The taskbar icon right-click menu SHALL expose an Exit action in every recording state.
When Meeting Assistant is idle or only processing older stopped meetings, the menu SHALL allow starting a meeting recording for each configured launch profile.
When a meeting is actively recording, the menu SHALL allow stopping the recording and continuing transcription/summary generation.
During an active recording, the normal stop action SHALL be labeled `Finish meeting` and SHALL be the only action in a dedicated menu section immediately below the `Open agent` section.
During an active recording, microphone selection, cancel/discard, and profile-switch actions SHALL appear in a separate fine-grained controls section below `Finish meeting`.
When a meeting is actively recording, the menu SHALL allow canceling the recording and discarding that run's artifacts.
When a meeting is actively recording, the menu SHALL allow switching to each configured launch profile other than the current active profile.
Selecting Exit while Meeting Assistant is idle SHALL stop the application without an additional confirmation prompt.
Selecting Exit while Meeting Assistant is recording, transcribing, recognizing speakers, or summarizing SHALL show a confirmation dialog before stopping the application.
#### Scenario: Idle tray menu can start configured profiles
- **GIVEN** launch profiles `default` and `english` are configured
- **AND** no meeting recording is active
- **WHEN** the taskbar menu is opened
- **THEN** it offers start recording actions for `default` and `english`
#### Scenario: Recording tray menu prioritizes finishing the meeting
- **GIVEN** launch profiles `default` and `english` are configured
- **AND** a meeting is actively recording with profile `default`
- **WHEN** the taskbar menu is opened
- **THEN** `Finish meeting` is the only action in the section immediately below `Open agent`
- **AND** microphone selection, cancel/discard, and switching to `english` appear in a separate following section
- **AND** the menu does not offer switching to `default`
#### Scenario: Active recording has priority over older summarizing runs
- **GIVEN** an older meeting is still summarizing
- **WHEN** a newer meeting is actively recording
- **THEN** the taskbar icon indicates recording
#### Scenario: Tray menu always exposes Exit
- **GIVEN** Meeting Assistant is running
- **WHEN** the taskbar menu is opened
- **THEN** it offers an Exit action
#### Scenario: Idle Exit stops immediately
- **GIVEN** no recording, transcription, speaker recognition, or summary work is running
- **WHEN** the user selects Exit from the taskbar menu
- **THEN** Meeting Assistant stops the application without an additional confirmation prompt
#### Scenario: In-progress Exit asks for confirmation
- **GIVEN** Meeting Assistant is recording, transcribing, recognizing speakers, or summarizing
- **WHEN** the user selects Exit from the taskbar menu
- **THEN** Meeting Assistant asks for confirmation before stopping the application
@@ -0,0 +1,15 @@
## 1. Tray Menu Behavior
- [x] 1.1 Add a failing behavior test proving that an active recording labels the normal stop action `Finish meeting`, places it alone immediately below `Open agent`, and keeps fine-grained controls in the following section.
- [x] 1.2 Add explicit section metadata to the tray-menu model, reorder the active-recording actions, and render separators from that metadata.
## 2. Verification
- [x] 2.1 Review the touched menu builder and renderer for DRYness, SOLID design, and simplicity while preserving behavior.
- [x] 2.2 Run focused taskbar-menu tests, the Windows application build, the full solution tests, and strict OpenSpec validation.
## 3. Refactor Follow-up
- [x] 3.1 Lock down idle section boundaries and active-recording layout when no microphone is available.
- [x] 3.2 Remove the tray-menu section helper's hidden input mutation without changing rendered behavior.
- [x] 3.3 Run focused and full verification, then validate the OpenSpec change strictly.
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-03
@@ -0,0 +1,60 @@
## Context
The Windows microphone source currently creates one NAudio `IWaveIn` for the lifetime of a recording. When the endpoint is unplugged, NAudio reports a WASAPI exception through `RecordingStopped`; the source completes exceptionally, the composite source treats that as fatal, and `MeetingRecordingCoordinator` ends the run. The composite source already tolerates a temporarily quiet microphone by mixing system audio with synthetic silence after its alignment timeout, so recovery can be isolated to the microphone side.
The microphone selection provider already re-enumerates active endpoints whenever it creates a capture. Its selection rules ignore an unavailable runtime/configured device and fall back to the current Windows default. The missing behavior is retrying that resolution after an active capture fails.
## Goals / Non-Goals
**Goals:**
- Keep the active meeting run alive when microphone capture fails or stops unexpectedly.
- Re-resolve the effective microphone on every recovery attempt so another active endpoint can take over.
- Keep system-loopback audio flowing while microphone recovery is pending.
- Verify recovery deterministically through the public audio-source contract without physical audio devices.
**Non-Goals:**
- Recover system-loopback capture failures.
- Persist or change the user's runtime microphone selection.
- Add UI, endpoint, or configuration controls for recovery.
- Splice or manufacture microphone audio for the disconnected interval.
## Decisions
### Keep retry orchestration outside the NAudio adapter
`MicrophoneAudioSource` will own a recovery loop and ask `IMicrophoneDeviceProvider` for a new capture source on each attempt. The Windows provider will continue to own endpoint enumeration and selection, while an NAudio-specific adapter will own one `IWaveIn` lifetime.
This keeps device selection and WASAPI details behind a narrow boundary and makes the observable recovery behavior testable with deterministic capture sources. Retrying the same `IWaveIn` instance was rejected because a disconnected WASAPI client is not a reliable basis for endpoint failover.
### Treat unexpected completion and capture exceptions as recoverable
While the recording cancellation token remains active, microphone-source creation failures, capture exceptions, and clean-but-unexpected capture completion will all trigger another attempt. Cancellation remains the only normal terminal condition for the microphone stream.
This deliberately contains microphone failures without changing the composite source's handling of system-audio failures.
### Re-resolve after a bounded delay
Each recovery attempt will call the provider again after a short fixed delay. Recreating through the provider re-enumerates active devices and applies the existing runtime selection, configured selection, and Windows-default fallback rules. The delay prevents a busy loop while Windows is still updating endpoint state.
No new setting is introduced because recovery timing is an internal reliability detail and does not need user tuning for the current scope.
### Reuse the composite source's missing-stream behavior
The recovering microphone enumerable remains active between attempts instead of completing. The independently pumped system source therefore continues writing chunks, and the composite source's existing alignment timeout mixes those chunks with silent microphone samples until real microphone chunks resume.
## Risks / Trade-offs
- **Windows endpoint enumeration can lag behind physical disconnects** → Retry through fresh provider calls until the device list and default endpoint stabilize.
- **A persistent microphone or driver failure can retry indefinitely** → Use a delay, log each failed attempt, and stop immediately when the recording is canceled.
- **The replacement endpoint can have different native capabilities** → Continue requesting the run's configured PCM format through the same NAudio adapter; failed formats remain recoverable and retryable.
- **There is an unavoidable microphone gap during failover** → Preserve the meeting and system audio rather than inventing microphone samples; the mixed stream contains silence for the missing microphone interval.
## Migration Plan
No data or configuration migration is required. Deploy the updated executable normally. Rollback consists of restoring the previous executable; existing meeting artifacts are unaffected.
## Open Questions
None for this change.
@@ -0,0 +1,26 @@
## Why
Unplugging the active microphone currently propagates a WASAPI capture error through the recording pipeline and terminates the active meeting recording. Recording must remain available through transient device changes so that already-captured meeting work and continued system audio are not lost.
## What Changes
- Recover microphone capture when the active Windows capture endpoint disappears or otherwise stops unexpectedly.
- Re-resolve the effective microphone for each recovery attempt so an available configured, runtime-selected, default, or fallback endpoint can take over.
- Keep the active recording and its independent system-audio capture alive while no microphone is temporarily available.
- Log microphone recovery failures and successful capture restarts without terminating the meeting run.
## Capabilities
### New Capabilities
None.
### Modified Capabilities
- `meeting-recording`: Active recording becomes resilient to microphone endpoint disconnection and automatically resumes microphone capture from an available endpoint.
## Impact
- Affects the Windows microphone capture source and device-provider boundary.
- Adds behavior tests around the public meeting audio-source contract.
- Does not change recording endpoints, tray controls, system-loopback capture, or transcription-provider APIs.
@@ -0,0 +1,125 @@
## MODIFIED Requirements
### Requirement: Recording mode captures microphone and computer output
Meeting Assistant SHALL capture microphone input and computer output and combine them into one audio stream for transcription.
Meeting Assistant SHALL capture audio as 16 kHz mono PCM chunks for the existing recording and transcription pipeline.
Meeting Assistant SHALL capture microphone and system loopback as separate input streams before producing the final mono chunks.
Meeting Assistant SHALL clean the microphone stream with a local acoustic echo cancellation stage that uses system loopback as the far-end reference.
Meeting Assistant SHALL produce final mono chunks by adding the cleaned microphone samples and system samples.
Meeting Assistant SHALL align microphone and system samples through per-source buffers before mixing and SHALL NOT emit normal live audio chunks that contain only one source while the other source is merely delayed.
When one source stays quiet beyond the alignment timeout, Meeting Assistant SHALL mix the available source with synthetic silence for the missing source instead of blocking transcription.
Meeting Assistant SHALL allow the final microphone/system mono mix to apply configurable microphone and system gain before combining samples.
Meeting Assistant SHALL use the active run or launch profile recording options when configuring capture format and final microphone/system gains.
Meeting Assistant SHALL clamp mixed samples after gain is applied.
Meeting Assistant SHALL write only the mixed stream to the temporary WAV used by transcription and finalization.
Meeting Assistant SHALL allow `Recording:MicrophoneDeviceId` to select a Windows microphone capture endpoint.
When `Recording:MicrophoneDeviceId` is blank or absent, Meeting Assistant SHALL use the Windows default capture endpoint.
When a microphone is selected from the tray icon menu, Meeting Assistant SHALL use that selected microphone for later recording starts until another microphone is selected or the process exits.
The tray icon right-click menu SHALL expose a `Microphone` submenu listing active microphone capture endpoints.
The `Microphone` submenu SHALL mark exactly one effective microphone as checked.
When no runtime microphone override is selected, the checked microphone SHALL be the configured microphone when it is available, otherwise the Windows default capture endpoint.
When the active microphone endpoint disappears, microphone capture fails, or microphone capture stops unexpectedly while a meeting recording is active, Meeting Assistant SHALL keep the meeting recording active and SHALL repeatedly re-resolve and restart microphone capture until capture succeeds or the recording is stopped.
Each microphone recovery attempt SHALL re-enumerate active microphone endpoints and apply the existing runtime-selected, configured, and Windows-default selection rules so an available endpoint can take over.
While microphone recovery is pending, Meeting Assistant SHALL keep system-loopback capture active and SHALL continue producing mixed audio with synthetic silence for the missing microphone stream.
#### Scenario: Both sources produce audio
- **WHEN** microphone and computer output audio chunks are available
- **THEN** Meeting Assistant mixes them into one PCM stream before transcription
#### Scenario: Mixed audio uses cleaned microphone and system audio
- **GIVEN** the echo canceller cleans a microphone chunk to sample `2000`
- **AND** the matching system chunk has sample `10000`
- **WHEN** microphone and system chunks are mixed with gains `1` and `1`
- **THEN** the mixed sample is `12000`
#### Scenario: Temporary recording stores only the mixed stream
- **GIVEN** Meeting Assistant has mixed microphone and system audio into one PCM chunk
- **WHEN** Meeting Assistant appends the chunk to the temporary recording
- **THEN** the main temporary WAV contains the mixed PCM
- **AND** no microphone or system sidecar WAV is written
#### Scenario: Launch profile recording options configure capture and gains
- **GIVEN** an active launch profile configures sample format and microphone/system mix gains
- **WHEN** Meeting Assistant captures and mixes audio for that run
- **THEN** the microphone and system capture sources receive that launch profile recording configuration
- **AND** the mixed output uses that launch profile's microphone/system gains
#### Scenario: Delayed sources are buffered before mixing
- **GIVEN** microphone audio arrives before matching system audio
- **WHEN** matching system audio arrives after a short delay
- **THEN** Meeting Assistant emits one mixed chunk for the aligned samples
- **AND** it does not emit separate microphone-only and system-only chunks for that delayed pair
#### Scenario: Quiet system audio does not block microphone transcription
- **GIVEN** microphone audio arrives
- **AND** system loopback audio does not arrive within the alignment timeout
- **WHEN** Meeting Assistant mixes the available audio
- **THEN** it emits the microphone audio mixed with silent system audio
- **AND** live transcription can continue while system loopback is quiet
#### Scenario: Continuous microphone audio does not suppress the alignment timeout
- **GIVEN** microphone audio keeps arriving
- **AND** system loopback audio stays unavailable past the alignment timeout
- **WHEN** Meeting Assistant checks the buffered microphone audio
- **THEN** it emits the buffered microphone audio mixed with silent system audio
- **AND** it does not wait indefinitely for a loopback chunk
#### Scenario: Device-level capture cannot be verified in tests
- **WHEN** automated tests run without live audio devices
- **THEN** Meeting Assistant verifies the audio mixer through deterministic source abstractions rather than depending on physical microphone or speaker devices
#### Scenario: Configured microphone is used for capture
- **GIVEN** `Recording:MicrophoneDeviceId` identifies an active microphone endpoint
- **WHEN** Meeting Assistant starts microphone capture
- **THEN** it captures from that endpoint
#### Scenario: Blank microphone setting uses Windows default
- **GIVEN** `Recording:MicrophoneDeviceId` is blank
- **WHEN** Meeting Assistant starts microphone capture
- **THEN** it captures from the Windows default capture endpoint
#### Scenario: Tray menu lists microphones with current selection checked
- **GIVEN** active microphone endpoints `integrated microphone` and `other microphone`
- **AND** `integrated microphone` is the effective microphone
- **WHEN** the taskbar menu is opened
- **THEN** it shows a `Microphone` submenu
- **AND** the `integrated microphone` item is checked
- **AND** the `other microphone` item is unchecked
#### Scenario: Tray microphone selection changes later capture
- **GIVEN** active microphone endpoints `integrated microphone` and `other microphone`
- **WHEN** the user selects `other microphone` from the taskbar microphone submenu
- **THEN** later recording starts capture from `other microphone`
#### Scenario: Disconnected microphone fails over during recording
- **GIVEN** a meeting is actively recording from one microphone and another microphone is available
- **WHEN** the active microphone is disconnected and its capture fails
- **THEN** the meeting recording remains active
- **AND** Meeting Assistant re-resolves the effective microphone and resumes capture from the available microphone
#### Scenario: Recording continues while no microphone is available
- **GIVEN** a meeting is actively recording
- **WHEN** the active microphone disconnects and no microphone is temporarily available
- **THEN** Meeting Assistant keeps the meeting recording and system-loopback capture active
- **AND** emits system audio mixed with synthetic microphone silence
- **WHEN** a microphone becomes available
- **THEN** Meeting Assistant resumes microphone capture for the same meeting run
@@ -0,0 +1,14 @@
## 1. Microphone recovery behavior
- [x] 1.1 Add a failing behavior test proving active microphone capture moves to a newly resolved capture source after the current source fails.
- [x] 1.2 Refactor the microphone device-provider boundary so recovery orchestration is platform-independent and individual NAudio capture lifetimes remain Windows-specific.
- [x] 1.3 Implement bounded-delay microphone recovery that re-resolves devices after creation failures, capture failures, and unexpected capture completion until recording cancellation.
- [x] 1.4 Add coverage proving capture recovers when no microphone is initially available and a later resolution succeeds.
- [x] 1.5 Add a failing selection test and fall back to an active endpoint when the selected and Windows-default endpoints are unavailable.
## 2. Verification
- [x] 2.1 Refactor the touched capture path for DRYness, SOLID boundaries, and KISS while preserving behavior.
- [x] 2.2 Run the focused microphone-selection and audio-source behavior tests plus the Windows application build.
- [x] 2.3 Run the full solution test suite and `openspec validate recover-microphone-disconnect --strict`.
- [x] 2.4 Verify the local health and recording-status surfaces without interrupting an active meeting run.
+51 -76
View File
@@ -44,6 +44,8 @@ The meeting note frontmatter SHALL link to the transcript, assistant context, an
Generated artifact notes SHALL link only to the other notes from the same run and SHALL omit the frontmatter property that would reference themselves.
Meeting Assistant SHALL escape generated meeting-note frontmatter string values after metadata enrichment and workflow rules have been applied, immediately before writing the final markdown file.
#### Scenario: Meeting note links to generated artifacts
- **WHEN** Meeting Assistant creates a meeting note
- **THEN** the note frontmatter links to the configured transcript, assistant context, and summary note locations
@@ -59,6 +61,12 @@ Generated artifact notes SHALL link only to the other notes from the same run an
- **THEN** the artifact frontmatter links to the other run notes
- **AND** the artifact frontmatter omits the property for the artifact's own note type
#### Scenario: Generated frontmatter remains parseable after attendee transforms
- **GIVEN** meeting metadata or workflow rules produce an attendee name that contains a single quote
- **WHEN** Meeting Assistant writes the final meeting note
- **THEN** the attendee value is escaped in frontmatter
- **AND** the meeting note frontmatter remains parseable when read back
### Requirement: Meeting notes preserve user-authored content
Meeting Assistant SHALL preserve user-authored meeting notes and include them as input when generating summaries, decisions, and next steps.
@@ -221,95 +229,43 @@ Meeting Assistant SHALL expose a diagnostic endpoint that reloads application co
- **AND** future workflow events use the reloaded workflow automation configuration
### Requirement: Meeting automation rules support lifecycle triggers
Meeting Assistant SHALL support rule triggers for `created`, `state_transition`, `speaker_identified`, and `transcript_line`.
Meeting Assistant SHALL support rule triggers for `created`, `state_transition`, `speaker_identified`, `transcript_line`, and `attendee_added`.
A `state_transition` trigger MAY filter by `from`, `to`, or both state values.
An `attendee_added` trigger MAY filter by `equals`, `contains`, or `regex` against the attendee value being added. `equals` and `contains` filters SHALL match case-insensitively.
A `speaker_identified` trigger MAY filter by speaker name.
Meeting Assistant SHALL apply attendee-added transformations before storing attendees added from meeting metadata, accepted recording-prompt metadata, workflow `add_attendee` steps, screenshot OCR, speaker identity updates, and summarizer add-attendee tools.
A `transcript_line` trigger MAY filter by speaker name.
#### Scenario: State transition rule matches from and to
- **GIVEN** a configured rule that triggers on a state transition from `collecting metadata` to `transcribing`
- **WHEN** Meeting Assistant transitions that meeting from `collecting metadata` to `transcribing`
- **THEN** it applies the rule steps
#### Scenario: Speaker identified rule filters by name
- **GIVEN** a configured rule that triggers when speaker `Ada` is identified
- **WHEN** Meeting Assistant identifies speaker `Grace`
- **THEN** it does not apply the rule
- **WHEN** Meeting Assistant identifies speaker `Ada`
- **THEN** it applies the rule steps
#### Scenario: Transcript line rule rewrites masked profanity before persistence
- **GIVEN** a configured rule that triggers on transcript line writes and sets `transcript.line` by replacing `*****` with `[redacted]`
- **WHEN** Meeting Assistant writes a transcript line for speaker `Guest-1` containing `*****`
- **THEN** the written transcript line contains `[redacted]`
- **AND** the written transcript line does not contain `*****`
#### Scenario: Attendee added rule filters and transforms a metadata attendee
- **GIVEN** a configured rule that triggers on added attendees containing `@contoso.com`
- **AND** the rule sets `attendee.name` to a display name derived from the added attendee
- **WHEN** Meeting Assistant adds attendee `Ada Lovelace <ada@contoso.com>` from meeting metadata
- **THEN** the stored meeting attendee is the transformed attendee name
- **AND** the original attendee string is not stored
### Requirement: Meeting automation rules support conditions and steps
Meeting Assistant SHALL support rule conditions using an expression engine.
Meeting Assistant SHALL expose the added attendee name to `attendee_added` rule conditions and Razor step templates as `attendee.name`.
Rules SHALL support nested `and`, `or`, and `not` condition groups.
The `set_property` step SHALL support setting `attendee.name` during `attendee_added` events.
Step values SHALL support Razor syntax against the current meeting event model.
Rules with an `attendee_added` trigger SHALL be limited to transforming `attendee.name` with `set_property`.
Step values SHALL treat `@` characters inside valid email address tokens as literal text rather than Razor transitions.
Direct meeting note file writes SHALL NOT emit `attendee_added` or transform attendee values.
Meeting Assistant SHALL expose the formatted transcript line and transcript speaker to `transcript_line` rule conditions and Razor step templates.
Meeting Assistant SHALL support these initial rule steps:
- `add_attendee`
- `remove_attendee`
- `set_property`
- `add_context`
- `add_project`
The `set_property` step SHALL support setting `transcript.line` during `transcript_line` events.
#### Scenario: Nested conditions choose a matching rule
- **GIVEN** a configured rule with nested `and`, `or`, and `not` conditions over meeting title, attendees, and event data
- **WHEN** the condition evaluates to true
- **THEN** Meeting Assistant applies the rule
- **WHEN** the condition evaluates to false
- **THEN** Meeting Assistant skips the rule
#### Scenario: Templated context mentions identified speaker
- **GIVEN** a configured `speaker_identified` rule with an `add_context` step using Razor syntax
- **WHEN** Meeting Assistant identifies matching speaker `Ada`
- **THEN** it appends rendered context text containing `Ada` to the assistant context note
#### Scenario: Email addresses do not trigger Razor templating
- **GIVEN** a configured rule step value containing `Support@contoso.com`
- **WHEN** the rule runs
- **THEN** Meeting Assistant preserves the email address as literal text
#### Scenario: Email addresses can appear beside Razor templating
- **GIVEN** a configured rule step value containing both `Support@contoso.com` and a Razor expression
- **WHEN** the rule runs
- **THEN** Meeting Assistant renders the Razor expression and preserves the email address as literal text
#### Scenario: Rule can clean a meeting title
- **GIVEN** a configured state-transition rule that matches a title containing a configured marker
- **WHEN** the rule runs
- **THEN** Meeting Assistant can update the meeting title through `set_property`
#### Scenario: Transcript line conditions can use the written line and speaker
- **GIVEN** a configured `transcript_line` rule with conditions over `transcript.line` and `transcript.speaker`
- **WHEN** Meeting Assistant writes a transcript line that matches both conditions
- **THEN** it applies the rule steps before the line is persisted
#### Scenario: Attendee added rules can use regex and Razor
- **GIVEN** a configured `attendee_added` rule with a regex filter over the added attendee
- **WHEN** Meeting Assistant adds an attendee matching the regex
- **THEN** the rule can set `attendee.name` with a Razor template using `Model.Attendee.Name`
### Requirement: Workflow rules and speaker identities can be edited through a tray-launched assistant
Meeting Assistant SHALL expose an `Edit rules and identities` item from the tray icon menu.
Meeting Assistant SHALL expose an `Open agent` item from the tray icon menu.
The tray icon SHALL be implemented through the Uno notification icon stack.
The tray icon menu SHALL show every configured launch profile when recording can be started and SHALL include each profile's configured toggle hotkey in the corresponding start or switch menu item.
When the user selects `Edit rules and identities`, Meeting Assistant SHALL open a basic MewUI chat window for editing the configured workflow rules file and speaker identities.
When the user selects `Open agent`, Meeting Assistant SHALL open a basic MewUI chat window for editing the configured workflow rules file and speaker identities.
The chat window SHALL be titled `Edit rules and identities`, SHALL display user and assistant messages as visually distinct cards, SHALL display basic markdown emphasis, inline code, fenced code blocks, pipe tables rendered as grid views, and line breaks in agent responses, SHALL display a plain `Thinking...` line while the agent is working, MAY replace that line with `Reconnecting...` while retrying a transient agent request, SHALL provide a multiline text input at the bottom with placeholder text for asking to make a rule or list identities, SHALL send on Enter, SHALL insert a newline on Shift+Enter, and SHALL provide an explicit Send button.
The chat window SHALL be titled `Meeting Summary Agent`, SHALL display user and assistant messages as visually distinct cards, SHALL display basic markdown emphasis, inline code, fenced code blocks, pipe tables rendered as grid views, and line breaks in agent responses, SHALL display lightweight visible reasoning summary lines and tool-call lines that include only the called tool name while the agent is working, SHALL display a plain `Thinking...` line as the bottom activity line while the agent is working, MAY replace that bottom line with `Reconnecting...` while retrying a transient agent request, SHALL replace per-turn activity after completion with a collapsible `Worked for <duration>` expander between the user and assistant cards, SHALL show visible reasoning summary lines, tool-call lines, and status lines in order inside the expander when expanded, SHALL collapse those details again when the expander is closed, SHALL provide a multiline text input at the bottom with placeholder text for asking to make a rule or list identities, SHALL send on Enter, SHALL insert a newline on Shift+Enter, and SHALL provide an explicit Send button.
When a new chat message or thinking state is appended, the chat window SHALL scroll to the bottom of the newly rendered content if the conversation was already near the bottom or did not need scrolling before the append.
@@ -335,8 +291,8 @@ Meeting Assistant SHALL provide a diagnostic endpoint that opens the workflow ru
#### Scenario: Tray menu opens the editor
- **WHEN** the user opens the tray icon menu
- **THEN** the menu includes `Edit rules and identities`
- **WHEN** the user selects `Edit rules and identities`
- **THEN** the menu includes `Open agent`
- **WHEN** the user selects `Open agent`
- **THEN** Meeting Assistant opens the workflow rules and identities editor chat window
#### Scenario: Tray menu shows configured profile hotkeys
@@ -385,9 +341,16 @@ Meeting Assistant SHALL provide a diagnostic endpoint that opens the workflow ru
- **GIVEN** the rules editor chat window is open
- **WHEN** the user types a prompt and presses Enter
- **THEN** the user message appears in the conversation
- **AND** a plain `Thinking...` line appears while the agent is running
- **AND** a plain `Thinking...` line appears at the bottom while the agent is running
- **WHEN** the agent emits visible reasoning summaries or calls tools
- **THEN** lightweight activity lines appear above the bottom activity line with the visible reasoning summary text or only the called tool names
- **WHEN** the agent turn completes
- **THEN** the activity is represented between the user and assistant cards as a collapsed `Worked for <duration>` expander
- **AND** expanding it shows the visible reasoning summary, status, and tool-call lines in order
- **AND** collapsing it hides those details again
- **AND** if no visible reasoning summary was emitted, the expander includes the fallback `Thinking...` line
- **AND** the first model request for that turn is marked as user-initiated
- **AND** the final assistant response replaces the thinking line
- **AND** the final assistant response replaces the live activity lines
#### Scenario: Rules editor displays agent request failures
- **GIVEN** the rules editor chat window is open
@@ -400,3 +363,15 @@ Meeting Assistant SHALL provide a diagnostic endpoint that opens the workflow ru
- **WHEN** a new user or assistant message is appended
- **THEN** the conversation scrolls to the bottom of the newly rendered message content
### Requirement: Interactive agent uses assistant context as meeting memory
The interactive agent window instructions SHALL identify each assistant context file as meeting-specific memory.
When the user asks the interactive agent to fix or investigate a meeting or summary, the instructions SHALL direct the agent to read the matching assistant context for clues about problems, missing information, assumptions, prior fixes, and conclusions.
After repairing a meeting or summary, the instructions SHALL direct the agent to append a concise record of its fixes and conclusions to the matching assistant context.
#### Scenario: Interactive agent repairs a meeting artifact
- **GIVEN** a meeting has an assistant context file
- **WHEN** the user asks the interactive agent to fix that meeting or its summary
- **THEN** the agent instructions direct it to inspect the matching assistant context for relevant meeting-specific memory
- **AND** direct it to append its fixes and conclusions to that assistant context
+46
View File
@@ -140,3 +140,49 @@ The summary-agent instructions SHALL tell the agent to keep the one-line summary
- **THEN** Meeting Assistant refuses the write
- **AND** does not mark the summary as written
### Requirement: Summary agent preserves meeting-specific working memory
When Meeting Assistant uses the built-in summary-agent instructions, those instructions SHALL identify assistant context as persistent meeting-specific memory.
The built-in instructions SHALL direct the summary agent to append unexpected problems, missing information, and assumptions encountered while summarizing to assistant context.
#### Scenario: Summarizer encounters uncertainty
- **GIVEN** the built-in summary-agent instructions are in use
- **WHEN** the summary agent encounters an unexpected problem, cannot find needed information, or must make an assumption
- **THEN** its instructions direct it to append a concise record to the meeting's assistant context
- **AND** later agents can discover that record when working on the same meeting
### Requirement: Summary agents tolerate Responses event streams and malformed tool arguments
Meeting Assistant SHALL consume successful summary-agent Responses results through the supported OpenAI Responses and Agent Framework Server-Sent Events adapter.
When a Responses event stream delivers completed output items separately from the final response metadata, Meeting Assistant SHALL assemble those output items into one agent response while preserving final response metadata and usage.
Meeting Assistant SHALL provide an agent setting that selects streaming or non-streaming Responses transport. Streaming SHALL be enabled by default. When streaming is disabled, Meeting Assistant SHALL use the supported non-streaming OpenAI Responses client and adapter.
When a returned function call contains arguments that are not a valid JSON object, Meeting Assistant SHALL NOT invoke the requested function and SHALL return an invalid-tool-arguments result associated with the original call ID to the agent so it can correct the call.
When Meeting Assistant translates chat messages into a Responses request, every message input item SHALL retain the `message` item discriminator required by the OpenAI SDK and Responses API.
#### Scenario: Streamed text response is assembled
- **WHEN** the configured Responses endpoint returns completed message output in Server-Sent Events followed by final response metadata
- **THEN** the summary agent receives the completed message text, response metadata, and usage without a JSON document parse failure
#### Scenario: Streamed function call is assembled
- **WHEN** the configured Responses endpoint returns a completed function-call output item in Server-Sent Events
- **THEN** the summary agent receives the function call with its call ID, function name, and parsed arguments
#### Scenario: Streaming transport can be disabled
- **WHEN** `MeetingAssistant:Agent:UseStreaming` is `false`
- **THEN** the summary agent requests a non-streaming Responses result
- **AND** converts the response through the supported OpenAI Responses adapter
#### Scenario: Chat message input remains a Responses message
- **WHEN** the summary agent sends a user or assistant chat message through the Responses client
- **THEN** the outbound Responses input item has type `message`
- **AND** the request does not contain an `unknown` input item type
#### Scenario: Malformed function arguments are returned to the agent
- **WHEN** the model returns a function call whose arguments are not a valid JSON object
- **THEN** Meeting Assistant does not invoke the requested function
- **AND** sends a function result with an invalid-tool-arguments error for the original call ID back to the agent
- **AND** allows the agent loop to continue