Public Access
35 lines
2.7 KiB
Markdown
35 lines
2.7 KiB
Markdown
## Context
|
|
|
|
`LiteLlmScreenshotOcrClient` currently builds and posts a raw Responses JSON payload, then parses the successful body as one JSON document. It inherits endpoint, model, and key values from `AgentOptions`, but never reads `AgentOptions.UseStreaming`. The summary and workflow agents already use `LiteLlmResponsesChatClient`, which selects the OpenAI SDK streaming or non-streaming Responses method and maps both through the Microsoft.Extensions.AI adapter.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Make screenshot OCR use `Agent:UseStreaming` without adding another setting.
|
|
- Reuse the supported Responses SDK transport and response adapter.
|
|
- Preserve screenshot-specific endpoint, model, key, prompt, image, crop, attendee, and timeout behavior.
|
|
- Keep non-streaming screenshot OCR working when streaming is disabled.
|
|
|
|
**Non-Goals:**
|
|
|
|
- Expose OCR token deltas to the UI or assistant context.
|
|
- Change screenshot OCR retry, crop, attendee, or note-block semantics.
|
|
- Add file upload or remote image URL support.
|
|
|
|
## Decisions
|
|
|
|
1. Route screenshot OCR through `LiteLlmResponsesChatClient` instead of maintaining a second Responses parser. The screenshot client will construct one user chat message containing prompt text and PNG `DataContent`, then consume the buffered `ChatResponse.Text`. This keeps transport selection, SDK request creation, SSE assembly, and non-streaming mapping in one client.
|
|
|
|
2. Extend the shared Responses message translator to map image `DataContent` to an `input_image` content block using its data URI. Text and image blocks remain in one `type: message` input item, matching the existing OCR payload shape.
|
|
|
|
3. Use `AgentOptions.UseStreaming` for screenshot OCR even when the screenshot-specific endpoint or model overrides are set. Endpoint/model/key remain independently overrideable; transport is an agent-wide behavior setting.
|
|
|
|
4. Disable reasoning and compaction for the one-turn OCR request, preserving the existing screenshot client behavior while still using the agent reconnection settings and selected transport.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Shared-client diagnostics mention summary context]** Some low-level logs are named for the summary pipeline. → Avoid passing summary compaction state and keep screenshot-specific completion/failure logs at the screenshot client boundary.
|
|
- **[Multimodal translation expands shared client scope]** Incorrect content mapping could affect summary requests. → Add request-body behavior coverage proving prompt and PNG data URI are preserved, while existing summary message tests protect text translation.
|
|
- **[Provider image support varies]** A configured model may reject image input. → Preserve the provider error and existing screenshot OCR failure/retry behavior.
|