meeting-assistant/openspec/changes/apply-streaming-to-screenshot-ocr/design.md

Context

LiteLlmScreenshotOcrClient currently builds and posts a raw Responses JSON payload, then parses the successful body as one JSON document. It inherits endpoint, model, and key values from AgentOptions, but never reads AgentOptions.UseStreaming. The summary and workflow agents already use LiteLlmResponsesChatClient, which selects the OpenAI SDK streaming or non-streaming Responses method and maps both through the Microsoft.Extensions.AI adapter.

Goals / Non-Goals

Goals:

• Make screenshot OCR use Agent:UseStreaming without adding another setting.
• Reuse the supported Responses SDK transport and response adapter.
• Preserve screenshot-specific endpoint, model, key, prompt, image, crop, attendee, and timeout behavior.
• Keep non-streaming screenshot OCR working when streaming is disabled.

Non-Goals:

• Expose OCR token deltas to the UI or assistant context.
• Change screenshot OCR retry, crop, attendee, or note-block semantics.
• Add file upload or remote image URL support.

Decisions

1. Route screenshot OCR through LiteLlmResponsesChatClient instead of maintaining a second Responses parser. The screenshot client will construct one user chat message containing prompt text and PNG DataContent, then consume the buffered ChatResponse.Text. This keeps transport selection, SDK request creation, SSE assembly, and non-streaming mapping in one client.

2. Extend the shared Responses message translator to map image DataContent to an input_image content block using its data URI. Text and image blocks remain in one type: message input item, matching the existing OCR payload shape.

3. Use AgentOptions.UseStreaming for screenshot OCR even when the screenshot-specific endpoint or model overrides are set. Endpoint/model/key remain independently overrideable; transport is an agent-wide behavior setting.

4. Disable reasoning and compaction for the one-turn OCR request, preserving the existing screenshot client behavior while still using the agent reconnection settings and selected transport.

Risks / Trade-offs

• [Shared-client diagnostics mention summary context] Some low-level logs are named for the summary pipeline. → Avoid passing summary compaction state and keep screenshot-specific completion/failure logs at the screenshot client boundary.
• [Multimodal translation expands shared client scope] Incorrect content mapping could affect summary requests. → Add request-body behavior coverage proving prompt and PNG data URI are preserved, while existing summary message tests protect text translation.
• [Provider image support varies] A configured model may reject image input. → Preserve the provider error and existing screenshot OCR failure/retry behavior.