← Pace

On-device field guide

Screen-aware AI assistant for Mac

Try a screen-aware Mac assistant with a read-only document workflow. Check permissions, local or cloud planning, answer accuracy and common failure cases.

A screen-aware AI assistant uses the state of the active Mac application to answer questions, guide the user, or perform an approved action. It should not treat a screenshot as unlimited authority. Reliable screen assistance combines the least invasive observation available with accessibility structure, visual grounding when necessary, and verification after each action.

Three kinds of screen context

  1. Accessibility structure: roles, labels, values, focus, and actions exposed by macOS. This is usually the best source for semantic controls.
  2. Visual context: pixels, OCR, or detected marks for applications where accessibility information is incomplete.
  3. Interaction state: the active window, recent user input, and the visible change after an action.

Each source has gaps. Accessibility can be incomplete, visual models can misread coordinates, and a visible change may not prove the intended result. A robust assistant cross-checks them and fails visibly when confidence is insufficient.

The action loop

A useful loop is observe, plan, request approval when required, act, and verify. For a click, verification might confirm a changed accessibility value or a new window state. For a workflow guide, the assistant should advance only after the user's click matches the expected state. A coordinate alone is not a durable outcome.

Pace's screen model

Pace can use local accessibility context, OCR, and a local vision model. Its click executor scores candidates, verifies the result, and has recovery paths for ambiguous targets. Watch mode records bounded local observations when enabled. Pace can also draw guidance on screen and wait for the user to complete the expected step. Screen and accessibility permissions remain controlled through macOS and Pace's own settings.

Privacy and limits

In local mode, screen analysis stays on the Mac. If a user selects an off-device planner tier, Pace shows the external state and records it in the local audit log; capability-specific privacy pins can still require local processing. Users should disable screen or journal sources they do not want and clear local history through settings.

A small screen-aware workflow to try

Start with a screen explanation that you can check yourself. Pace has a text input as well as voice; using text for this first attempt keeps microphone and hotkey troubleshooting separate from screen context.

  1. Open a non-sensitive document in TextEdit. Give it a clear heading and one short sentence whose wording you know. Keep the document visible.
  2. In Pace settings, enable Read My Screen. Review the Screen Recording and Accessibility access requested by macOS. Choose the planner you intend to use; an off-device planner requires consent and shows an amber state.
  3. Type: “Read the heading and summarize the visible sentence. Do not edit the document or take any actions.”
  4. Compare the answer with the text on screen. Confirm the document stayed unchanged. An answer about the wrong window or an invented sentence is a failed attempt, even if it sounds fluent.
  5. Change the sentence yourself and ask again. Check that the answer follows the new visible content rather than reusing the earlier description.

This is a suggested acceptance check, not a promise that every app or document is readable. The 0.3.20 usability record reports screen explanations without document edits in the signed app. It explicitly leaves voice accuracy and hotkey acceptance outside that non-voice pass.

When the screen answer is wrong or unavailable

  • No screen context: check Read My Screen and the macOS permissions first. A model being available does not mean it can see a window.
  • Wrong window or stale text: bring the intended document into view, send a fresh request and compare with the current sentence.
  • Local planner unavailable: inspect the selected model setup. Switching to an external planner changes the privacy boundary and needs an explicit choice.
  • A proposed click or edit: stop the read-only trial and inspect the separate action and approval workflow. Reading a screen does not authorize changing it.

Keep the first useful recurring task small: explaining a visible error, identifying a document heading or summarizing one paragraph. Once that works reliably in your own app, expand the workflow and verify each added action. Review the privacy controls before using sensitive material.

CTA: See how approved Mac actions work, review privacy controls, or download Pace.