fix: support streaming screenshot OCR

2026-08-03 16:14:01 +02:00
parent b9547ae4c4
commit 5d0ae84426
9 changed files with 362 additions and 145 deletions
@@ -0,0 +1,34 @@
+## Context
+
+`LiteLlmScreenshotOcrClient` currently builds and posts a raw Responses JSON payload, then parses the successful body as one JSON document. It inherits endpoint, model, and key values from `AgentOptions`, but never reads `AgentOptions.UseStreaming`. The summary and workflow agents already use `LiteLlmResponsesChatClient`, which selects the OpenAI SDK streaming or non-streaming Responses method and maps both through the Microsoft.Extensions.AI adapter.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Make screenshot OCR use `Agent:UseStreaming` without adding another setting.
+- Reuse the supported Responses SDK transport and response adapter.
+- Preserve screenshot-specific endpoint, model, key, prompt, image, crop, attendee, and timeout behavior.
+- Keep non-streaming screenshot OCR working when streaming is disabled.
+
+**Non-Goals:**
+
+- Expose OCR token deltas to the UI or assistant context.
+- Change screenshot OCR retry, crop, attendee, or note-block semantics.
+- Add file upload or remote image URL support.
+
+## Decisions
+
+1. Route screenshot OCR through `LiteLlmResponsesChatClient` instead of maintaining a second Responses parser. The screenshot client will construct one user chat message containing prompt text and PNG `DataContent`, then consume the buffered `ChatResponse.Text`. This keeps transport selection, SDK request creation, SSE assembly, and non-streaming mapping in one client.
+
+2. Extend the shared Responses message translator to map image `DataContent` to an `input_image` content block using its data URI. Text and image blocks remain in one `type: message` input item, matching the existing OCR payload shape.
+
+3. Use `AgentOptions.UseStreaming` for screenshot OCR even when the screenshot-specific endpoint or model overrides are set. Endpoint/model/key remain independently overrideable; transport is an agent-wide behavior setting.
+
+4. Disable reasoning and compaction for the one-turn OCR request, preserving the existing screenshot client behavior while still using the agent reconnection settings and selected transport.
+
+## Risks / Trade-offs
+
+- **[Shared-client diagnostics mention summary context]** Some low-level logs are named for the summary pipeline. → Avoid passing summary compaction state and keep screenshot-specific completion/failure logs at the screenshot client boundary.
+- **[Multimodal translation expands shared client scope]** Incorrect content mapping could affect summary requests. → Add request-body behavior coverage proving prompt and PNG data URI are preserved, while existing summary message tests protect text translation.
+- **[Provider image support varies]** A configured model may reject image input. → Preserve the provider error and existing screenshot OCR failure/retry behavior.