# Screen & vision How Pace sees the screen: cheap pixel diffing to know *when* to look, a local VLM to know *what's* there, OCR/AX to ground text and structure, and a click-recovery loop for when a guess misses. Foundational frameworks (ScreenCaptureKit, the speculative planner race) are covered in [`new-things.md`](new-things.md) and referenced here, not re-explained. --- ## Screen image diffing - What: A low-res grayscale fingerprint plus pixel-delta comparison that decides whether two screenshots differ "meaningfully" rather than pixel-identically. - Why here: The cheap pre-filter behind watch mode and the prewarm cache — avoids re-running the VLM or emitting an event for a blinking cursor or a clock tick. - Where: `PaceScreenImageDiffer.swift` — `PaceScreenImageDiffer.fingerprint(...)` and `.diff(...)`, producing a `PaceScreenVisualFingerprint` / `PaceScreenImageDiff`. - Source: internal — no external spec. ## Watch mode - What: An explicit watch-loop primitive that samples full-screen captures on an interval and emits typed events only past a diff threshold. - Why here: Powers "watch my screen" — `PaceScreenWatchModeController.startWatching(onMeaningfulChange:)` loops `CompanionScreenCaptureUtility.captureAllScreensAsJPEG()`, runs each capture through `PaceScreenWatchChangeDetector`, and only calls back (and journals) on a real `PaceScreenWatchEvent`. - Where: `PaceScreenWatchMode.swift` — `PaceScreenWatchModeController`, `PaceScreenWatchEventCategory` (`.majorScreenChange` / `.contentUpdate` / `.focusedRegionChange`). - Source: internal — no external spec. ## Set-of-Mark click recovery - What: When a coordinate click misses, render numbered marks over candidate elements and ask the model which numbered mark corresponds to the intended target, instead of re-guessing raw coordinates. - Why here: Pace's click-miss recovery path — `PaceSetOfMarkClickRecovery.resolve(inputs:renderMarks:groundMark:)` takes the failed click's screenshot + element map, renders marks, asks the VLM to pick a mark index, and turns that back into a screen coordinate. Pure decision logic — rendering and the VLM round-trip are injected closures, so it's unit-testable without AppKit or a live model. - Where: `PaceSetOfMarkClickRecovery.swift` — `PaceSetOfMarkClickRecovery.resolve`. - Source: https://arxiv.org/abs/2310.11441 (Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V) ## Set-of-Mark renderer - What: The drawing half of Set-of-Mark — takes a JPEG and a list of boxes, draws numbered outlines + index labels onto it. - Why here: Separates "what to draw" from "what to decide" — `PaceSetOfMarkRenderer.drawMarks(onJPEG:boxes:)` is the pure rendering function `PaceSetOfMarkClickRecovery` injects as its `renderMarks` closure. - Where: `PaceSetOfMarkRenderer.swift` — `PaceSetOfMarkRenderer.drawMarks(onJPEG:boxes:)`. - Source: internal — no external spec. ## Set-of-Mark mark-reading measurement - What: Set-of-Mark itself — numbering candidate regions on a screenshot so a VLM answers "which numbered mark" instead of emitting raw coordinates — is the technique behind both the click-recovery entries above; this entry is specifically about *measuring* how accurately a given VLM reads the marks. - Why here: Pace shipped Set-of-Mark click recovery without ever measuring the one number that determines whether it helps — how often the VLM picks the right mark index. `scripts/eval-vlm-mark-reading.py` closes that gap: it generates 119 synthetic Set-of-Mark cases (12 layouts × 5–15 elements) using `PaceSetOfMarkRenderer`'s exact drawing convention, so the eval measures the real rendering path, not an approximation of it. - Where: `scripts/eval-vlm-mark-reading.py` (mark-reading accuracy on synthetic layouts) and `scripts/eval-vlm-grounding.py` (real-VLM fixture eval against an AX-blind baseline, `capture-grounding-corpus.sh` builds the fixture corpus). As of 2026-07-12 both scripts exist and run against LM Studio, but per `PROJECT_STATUS.md`, first numbers are still pending an actual LM Studio run — treat any specific accuracy figure as not yet measured until that entry is updated. - Source: https://arxiv.org/abs/2310.11441 (same paper as Set-of-Mark click recovery above). ## Vision OCR client - What: A wrapper around Apple's Vision text-recognition request, run at the accurate recognition level, returning text plus bounding boxes. - Why here: Grounds "what does that text say" / "click the button labeled X" turns in real on-screen text rather than relying on the VLM's element-map guesses alone — fully on-device, ~50-200 ms per screenshot. - Where: `PaceVisionOCRClient.swift` — `PaceVisionOCRClient.recognizeText(...)`, built on `VNRecognizeTextRequest`. - Source: https://developer.apple.com/documentation/vision/vnrecognizetextrequest ## OCR data detector - What: Runs Foundation's `NSDataDetector` over OCR'd screen text to pull out typed entities — phone numbers, email addresses, URLs, dates, postal addresses. - Why here: Turns raw OCR text into actionable entities for turns like "add this date to my calendar" or "open that link" without an LLM call to extract them — the same trick TextEdit uses for its data-detector highlights, fully on-device. - Where: `PaceOCRDataDetector.swift` — `PaceDetectedEntity`, `PaceOCRDataDetector.detectEntities(in:)`. - Source: https://developer.apple.com/documentation/foundation/nsdatadetector ## OCR language resolver - What: A pure helper that intersects the user's `Locale.preferredLanguages` with Vision's supported OCR recognition languages, falling back to `en-US` if the intersection is empty. - Why here: Without it, OCR silently mis-reads non-English UIs (umlauts, kana, etc.) because Vision defaults to `en-US`; both language lists are injected, so this stays actor-free and unit-testable without a live `Locale` or Vision request. - Where: `PaceVisionOCRLanguageResolver.swift` — `PaceVisionOCRLanguageResolver.resolveRecognitionLanguages(preferredLanguagesFromLocale:supportedLanguagesFromVision:)`. - Source: internal — no external spec. ## Ambient context store - What: A permission-free snapshot of the user's current context — frontmost app name/bundle ID, window title, a lightweight AX-tree summary, and clipboard change count/timestamp. - Why here: Cheap situational grounding the planner gets on every turn without a screenshot or VLM call — `PaceAmbientContextStore` polls `NSWorkspace.shared.frontmostApplication` plus a background AX read and publishes a `PaceAmbientContextSnapshot` that renders into the planner prompt. - Where: `PaceAmbientContextStore.swift` — `PaceAmbientContextStore`, `PaceAmbientContextSnapshot`. - Source: internal — no external spec. ## Screen context service - What: The per-screen VLM analysis cache and AX+OCR+VLM coordinator, keyed by screen/display identity plus a pixel hash of the captured image. - Why here: Avoids paying VLM latency twice for the same unchanged screen — `PaceScreenContextService.prewarmScreenContext(reason:)` kicks off analysis speculatively at push-to-talk press (`.pushToTalkPress`) and at deeplink chat (`.deepLinkChat`); a pixel-hash match against `PaceCachedScreenAnalysis` reuses the cached element map instead of re-calling the VLM. - Where: `PaceScreenContextService.swift` — `PaceScreenContextService`, `PaceCachedScreenAnalysis`, `PaceScreenContextPrewarmReason`. - Source: internal — no external spec. ## Local VLM element maps - What: The client that sends a screenshot to the local VLM (LM Studio, OpenAI-compatible chat-completions) and parses back a structured element map — label, role, bounding box, and optional verbatim text per UI element. - Why here: This is what makes screen-action turns groundable in real coordinates instead of hallucinated ones — `LocalVLMClient.analyzeScreenshot(...)` builds the vision chat-completion request, and `groundMarkedClickTarget(...)` is the same client's Set-of-Mark grounding call. Both go through the loopback-only endpoint guard, never leaving the Mac. - Where: `LocalVLMClient.swift` — `LocalVLMClient` (conforms to `PaceScreenAnalysisClient`), `LocalVLMScreenElement`, `LocalVLMScreenAnalysis`. - Source: internal — loopback OpenAI-compatible endpoint; see LM Studio entry in [`new-things.md`](new-things.md). ## VLM-skip heuristic - What: A rule-based check on the transcript text that decides whether a turn is likely to reference the screen at all, before paying VLM latency. - Why here: Pure-knowledge turns ("what's the capital of France?") don't need a screenshot — `PaceTagParsers.transcriptIsLikelyScreenReferential` gates the VLM call so those turns skip straight to the text-only planner path; `AlwaysRunLocalVLMRegardlessOfTranscript=true` overrides it for testing. - Where: `PaceTagParsers.swift` — `PaceTagParsers.transcriptIsLikelyScreenReferential(_:)`. - Source: internal — no external spec. --- See also [`README.md`](README.md) for the full learning-roadmap index.