Pace — architecture (systems reference)

This is the canonical detailed architecture reference for Pace — the per-system bullet list that previously lived inline in the repository AGENTS.md. It was extracted to keep AGENTS.md a concise agent bootloader. The doctrine and the high-level constellation diagram live in overview.md; per-file responsibilities live in ../development/key-files.md.

App shape

App Type: Menu-bar-first (LSUIElement=true) with no Dock icon. On MacBooks with a camera housing, the always-available Living Notch signal and quick panel are joined by one on-demand resizable Command Center window. Displays without a reported housing use a conventional status item and standalone panel; Pace never draws a synthetic center notch.
iPad companion target: PacePad is a full-screen iPadOS 26 input/output surface in the same Xcode project. It owns Bonjour discovery, authenticated reconnect, tap-to-talk recording, local system TTS, the companion face, low-rate local presence detection, and requested-only camera still capture. It owns no planner, durable memory, Mac inspection, tool execution, or interruption policy.
Framework: SwiftUI (macOS native) with AppKit bridging for onboarding, the Command Center window, menu-bar panel, hardware-derived notch overlay, and cursor overlay
SwiftUI design system: The macOS app target links SaaSMakerUI at exact version 0.1.17. DS.palette brands its fonts and components with Pace’s semantic colors; SwiftUI roots apply the theme while cursor and glow geometry stay native. Packaging verifies the SaaSMakerUI_SaaSMakerUI.bundle font resources before signing.
Pattern: MVVM with @StateObject / @Published state management
Concurrency: @MainActor isolation, async/await throughout
Analytics: local-only no-op/timing-safe hooks via PaceAnalytics.swift; no cloud analytics SDK is linked.

iPad companion transport

PaceCompanionServer is a thin Mac adapter over the existing conversation, local audio transcription, privacy-pinned vision, world observation, and proactivity pipelines. It advertises _pace-companion._tcp and never creates a second planner or memory store.
PaceShared defines versioned messages and a length-prefixed frame containing bounded JSON control data plus optional AAC/JPEG bytes. Heartbeats and a bounded message-ID window handle connection health and replayed frames; shared policy also validates semantic payloads, stale sessions, privacy gates, heartbeat expiry, and bounded reconnect timing.
Pairing uses a six-digit, in-memory code only to establish TLS 1.3 PSK transport. The Mac then issues a random 256-bit credential stored in each device’s Keychain. Reconnect PSKs and a session HMAC derive from that credential; no secret is stored in UserDefaults or logged.
The iPad sends ordinary utterance audio and semantic presence events. JPEG bytes are sent only in response to an expiring camera_frame_request created by the conservative explicit physical-scene request parser. Pause or camera disable discards queued presence, cancels pending still continuations, and is re-checked before either side sends or accepts camera media. Temporary Mac audio files are removed after local transcription and requested images are never persisted.
If the Mac’s selected planner crosses the on-device boundary, the existing audit/tint behavior remains authoritative and the iPad processing and speaking states become amber for the same turn.

Planner

Planner: main screen/action planning uses the local OpenAI-compatible LM Studio reasoner. The current source default for both planner and fallback VLM is one resident qwen/qwen3.5-4b multimodal model, eliminating the missing-model and model-swap stalls caused by the previous separate Gemma/UI-Venus defaults. Pure-knowledge turns use the bundled in-process MLX planner when enabled; otherwise they use LM Studio’s native /api/v1/chat stream with reasoning: off and fall back before first output to the OpenAI-compatible route with both supported non-thinking hints. Structured action turns stay on Chat Completions for JSON-schema decoding. <think>…</think> stripping remains defensive. The bundled path prepares explicit system/user/assistant roles with a fresh token cache and a 2,048-token response limit; feature-specific JSON synthesis bypasses the agent planning scaffold. The pinned OSS ChatSession.streamResponse discards system instructions, so Pace uses the lower-level MLX generation API. The bundled path reuses a complete Hugging Face CLI asset cache and runs one real inference at launch; an early user turn awaits that same warmup task instead of racing a second model load. The 2026-08-27 qualification scored 16/19 on the legacy planner fixtures and 7/8 on production v10 action fixtures, with the focused five-turn set averaging 878 ms. Gemma 3 12B remains an optional quality-over-memory planner and qwen/qwen3-30b-a3b remains the stronger 48 GB swap. Apple Foundation Models is scoped to classification and structured extraction rather than world-knowledge answers. See ../development/info-plist-switches.md. Cloud planning is available only through explicitly consented tiers. First-launch tier default: on a brand-new install with no pace.planner.tier. UserDefaults state at all, PacePlannerTierStore.loadConfiguration() checks SystemLanguageModel.default.availability and resolves to .appleFoundationModels when Apple Intelligence is available, .local otherwise. Existing persisted tier choices remain authoritative.
Planner (BuddyPlannerClient): the active planner is chosen by the user in Settings → Planner via PacePlannerTier (Local / CLI bridge / CLI direct / Direct API / Apple FM). Default local model is qwen/qwen3.5-4b. BuddyPlannerClientFactory.makeDefault() dispatches on tier: LocalPlannerClient for .local and as a fallback for missing/invalid configs; AppleFoundationModelsPlannerClient for .appleFoundationModels; CloudBridgePlannerClient for .cliBridge; PaceLocalCLIPlannerClient for .cliDirect; and DirectAPIPlannerClient for .directAPI. The same CompanionSystemPrompt flows through every tier. Direct-API keys live in macOS Keychain via PaceKeychainStore. Any off-device tier tints the capsule amber and fails loud unless the user explicitly enables local fallback. The main action planner is decode-constrained to the v10 {spokenText,intent,payload} envelope; invalid structured output retries without prompt cache. Eval coverage lives in evals/fm-fixtures-actions/ and is scored on decoded actions.
- .cliDirect general brain (OpenSpec openspec/changes/archive/2026-07-12-codex-general-brain): a user-selectable tier that direct-spawns the user’s already-authenticated codex (default) or claude CLI — no Node bridge, only the CLI on PATH — as the brain for ALL turns (the same PaceLocalCLIPlannerClient the .research lane already uses). Upstream sub-selection persists via PacePlannerTierStore key pace.planner.tier.cliDirect.upstream (PaceLocalCLIUpstream, default .codex). Off-device, so it inherits the FULL cloud-bridge safety contract via a separate transport-aware consent: PaceCloudBridgeConsent.hasAcceptedDirectSpawnConsent() (canRunDirectSpawnTurn(now:), immediate foreground availability) — accepting the Node-bridge consent does NOT auto-grant the direct-spawn path. A .cliDirect turn tints the capsule amber (isOffDeviceTurnInFlight, keyed on plannerClientForThisTurn is PaceLocalCLIPlannerClient), writes a PaceAPIAuditLog entry (subsystem: "planner.cliDirect", target: <codex|claude>) so the Privacy dashboard flips off “0 bytes → X KB to codex”, and FAILS LOUD via PaceFailureNarrator on CLI error (no silent local fallback beyond the existing opt-in). Missing-binary preflight (PaceLocalCLIPlannerClient.isUpstreamBinaryOnPath) surfaces a plain-language “needs codex on PATH” hint before the turn. Codex is the phone-a-friend default: the .research lane defaults to Codex on fresh installs (PaceResearchTierStore.defaultCLIBridgeUpstream = .codex, defaultCLIBridgeModel = "" so Codex uses its own authenticated model; persisted upstreams honored for existing users), and cron/scheduled fires (CompanionManager+Lifecycle executeTaskCallback) route through a Codex direct-spawn brain ONLY when direct-spawn consent + 24h soak are satisfied (BuddyPlannerClientFactory.cronTaskBrainDecision, using the separate scheduled-turn gate) — otherwise they fall back to the local planner, so an unattended background fire never silently goes off-device (amber forced for the Codex path). The planner brain is also selectable from Settings → Models via a RAM-aware brain picker folded into PaceModelMemoryBudget’s budget section (PaceBundledModelsSettingsTab.memoryBudgetSection): picking a cloud/Apple-FM brain frees the local-planner RAM and the section live-updates the per-model breakdown, fits verdict, and largest-VLM-that-fits recommendation. Both the Planner tab and the Models picker route through the shared CompanionManager.selectPlannerTierWithConsent.

Voice, vision, TTS

Speech-to-Text: Apple SFSpeechRecognizer (on-device, requiresOnDeviceRecognition=true) is the default active provider — instant, no model download. TranscriptionProvider=whisperKit is accepted as a scaffolded local provider selection, but falls back to Apple Speech until the real WhisperKit streaming bridge lands. All cloud STT providers have been removed.
Text-to-Speech: two on-device conformers of BuddyTTSClient. The dependable default is LocalTTSClient (AVSpeechSynthesizer, Premium > Enhanced > compact LocalTTSVoiceIdentifier) because it has no sidecar or model-readiness dependency. TTSProvider=localServer opts into LocalServerTTSClient, backed by a loopback Kokoro-82M /v1/audio/speech service with per-utterance Apple fallback and a 30-second outage memo. Cloud TTS has been removed.
Local Vision-Language Model (optional): LM Studio at http://127.0.0.1:1234/v1 (OpenAI-compatible). The default qwen/qwen3.5-4b model is shared with the local planner, so only one model stays resident. Accessibility and native OCR run first; the VLM is the fallback for screens those native paths cannot expose. Planner/VLM HTTP roots are loopback-only. Pure-Q&A typed turns do not prewarm or call the VLM.
Screen Capture: ScreenCaptureKit (macOS 14.2+), multi-monitor support
Voice Input: Push-to-talk via AVAudioEngine + pluggable transcription-provider layer. System-wide keyboard shortcut via listen-only CGEvent tap.
Voice Input UI: PaceNativeInterfaceModel maps ready, listening, understanding, approval, acting, speaking, completed, blocked, and failed into one semantic state language. PaceSignalView renders that language across onboarding, the real notch, and the quick panel; off-device planner activity overrides the active signal to amber. Timeline rendering exists only for active states, and Reduce Motion uses static or short crossfade presentations.
Cursor: Codex-style arrow (CodexArrowShape) with linear-gradient fill, white highlight stroke, dual shadow.
Element Pointing: the planner embeds [POINT:x,y:label:screenN] tags in its response. The overlay parses these, maps coordinates to the correct monitor, and animates the cursor along a bezier arc to the target.

Memory

Two-tier in-context memory: every planner call carries the last K=4 turns verbatim (PaceThreadMemory.verbatimWindow()) AND a rolling summary of everything older (injectionPrefix(), rendered into the system prompt as <conversation_so_far>...</conversation_so_far>). The summary is refreshed by a detached Apple FM call after each turn via PaceThreadSummarizer; the user-facing path never blocks on summarization. Race-safety lives in a monotonic summaryVersion snapshot captured before each detached call, so out-of-order arrivals are dropped at applySummaryUpdate. In-session a 20-minute idle threshold still drops the live state (or Settings → Memory → Reset thread). The conversation now SURVIVES quit/relaunch: PaceThreadMemory.snapshot(now:)/restore(from:) are pure value-type accessors and PaceThreadMemoryStore persists a single on-device JSON (~/Library/Application Support/Pace/thread-memory.json, atomic write) after every turn/summary/reset; CompanionManager.start() rehydrates it. Policy is “resume always, until reset” — no staleness expiry; the file is wiped only on an explicit thread reset or when thread memory is disabled. The summary is still never journaled to paceHistory (only session id + lifecycle events are). Durable facts go through episodic memory instead.
Always-On Companion Mode (default off): typed provenance-bearing PaceWorldObservation evidence uses bounded atomic persistence, current-state/last-seen/change/presence queries, confidence decay, correction supersession, deterministic memory promotion, and a dedicated retrieval source. PaceCompanionModeController exposes off/starting/observing/interpreting/paused/degraded/privacy-blocked; PaceCompanionInterventionPolicy defaults to silent memory and keeps card/speech opt-ins separate. Settings → Companion and menu-bar indicators expose source consent, active capture, retention/storage, readiness, pause, and clear. App lifecycle starts explicitly enabled ambient/watch sources plus a production low-rate AVFoundation/Vision camera client that emits only non-identifying ephemeral person tracks and conservatively matches user-taught, locally persisted Vision feature prints across coarse camera zones. Teaching is an explicit centered-object capture; no photo is persisted. Ambient voice uses a bundled local pre-STT Core ML gate (PaceWakeWordClassifier, exact labels hey_pace and background); missing or malformed assets fail closed before Speech.framework is reachable. Accepted wakes hand off to the bounded push-to-talk conversation path. Silent cards and spoken interventions are separately default-off and wired through the intervention policy; speech additionally passes the existing live restraint/cooldown path. Routine promotion requires at least three unique supporting observations. On 2026-07-13 the owner directed “push through,” accepting the remaining unmeasured live/hardware and manual Cmd+R risk without representing those checks or documented thresholds as passed. See docs/product/companion-mode-privacy.md, docs/product/companion-mode-dogfood.md, and docs/product/pace-wake-word-classifier.md.

Actions, tools, integrations

Action Layer (agent mode): the planner should prefer <tool_calls> JSON blocks where the outer array is sequential steps and each inner array is a parallel group. Tool metadata lives in PaceToolRegistry so prompt docs, aliases, and risk labels share one source of truth; the local registry validates at app startup so schema examples, aliases, and enum coverage cannot silently drift. Legacy tags are still accepted: [CLICK:x,y], [DOUBLE_CLICK:x,y], [TYPE:text], [KEY:cmd+s], [SCROLL:up:3], [OPEN_APP:Safari], [OPEN_URL:https://example.com], [MUSIC:play], [VOLUME:up:2], [BRIGHTNESS:down:3], [CALENDAR:today], and [REMINDER:send invoice]. PaceActionTagParser extracts them; CompanionManager asks for user approval only for higher-risk non-undoable or external actions when the Approve Risky Actions preference is on, and the approval alert defaults to Cancel. Routine local actions such as click, scroll, window snap, app/URL open, media, volume, brightness, clipboard read, and undo can execute without the popup; blocking preflight issues still surface before execution. PaceActionApproval keeps the allow/cancel decision testable; PaceActionExecutor posts events after TTS playback starts. Single-clicks try PaceAXTargeter first (AX-tree press), falling back to CGEvent. AX set-value edits are logged in a session-local mutation log and can be restored with Undo.last / “undo that”. App launch/URLs use NSWorkspace; volume, brightness, media, and clipboard reads use local macOS APIs; Calendar/Reminders use EventKit for reads and creation. Mail recipient names can resolve through Contacts before composing a draft. Finder/Notes/Mail/Things/Shortcuts/Messages have first-pass local integrations. download_file downloads a user-named http(s) URL into ~/Downloads with URL validation, filename sanitization, and Finder-style collision suffixes — always approval-gated. Gated by EnableActions=true in Info.plist. Runtime smoke hooks are disabled unless PACE_ENABLE_SMOKE_HOOKS=1.
Trust surfaces (visible undo + reply replay + failure narration): every reversible mutation (createNote, appendNote, createReminder, createCalendarEvent, composeMail, createThingsToDo, runShortcut, downloadFile, recordFlow/runFlow, setTextValue/editSelectedText, openMessages with body, and generic mcp) raises a 5-second floating PaceUndoBanner next to the cursor; tapping it submits Undo.last through the executor. After every assistant turn finishes speaking, the notch panel renders a 30-second reply-replay button driven by the same post-processed string that flowed through TTS — replay does NOT re-stream the planner. Plain-language failures (planner offline, missing permission, click missed, sidecar TTS offline, MCP server not configured, cloud-bridge upstream error) are composed deterministically by PaceFailureNarrator (no LLM call) and routed through PaceRestraintGate.decide(...) so failure speech stays silent during a Zoom/active call. Reversibility uses the same set as PaceActionApprovalPolicy.actionIsReversibleMutation returns true for, and the click-missed narrator is wired to the all-fail click observation in PaceActionExecutor.
Typed automation catalog: Versioned manifests under Resources/automations/ contain canonical Pace tool calls, provenance, execution mode, required preferences, invocation examples, and ordered step groups. The indexed 17-entry starter library covers Notes templates, Calendar reads, timers, Finder locations, current-window layouts, bounded media control, weekly review, and end-of-day reset. PaceAutomationDefinitionLibrary validates every manifest at startup against PaceToolRegistry, rejects input injection, network/download, destructive, nested-flow, and opaque Shortcut capabilities, and calls PaceAutomationCompiler to reject partial parses. The compiler feeds the existing PaceActionTagParser, so typed automations reuse normal preflight, approval, executor, observation, and audit behavior. PaceAutomationCatalog builds a non-persisted view over typed automations (deterministicLocal), bounded Pace Programs (deterministicProgram), recorded flows (deterministicReplay), skills (plannerGrounded), and installed Shortcuts (externalOpaque). Explicit list/run commands still resolve locally by unique exact normalized name. Ordinary completed text permits implicit execution only for an exact authored invocation alias. Embedding similarity and local-model ambiguity resolution do not authorize execution; uncertain requests preserve the selected planner fallthrough. Catalog metadata is never added to any off-device planner prompt or conversation memory.
Programmable teachable skills: PaceProgramDefinition is a versioned declarative graph stored under ~/Library/Application Support/Pace/programs/. It supports ordered typed-action steps, conditions over one pre-run snapshot (weekday, local hour, or frontmost bundle identifier), and literal repetition. The whole graph—including inactive branches—is validated before save, discovery, and each run. Fixed limits are nesting depth 4, 50 source nodes, repeat count 10, and 50 worst-case expanded action steps. Programs cannot contain Lua, JavaScript, shell, imports, network/file APIs, raw Accessibility operations, or branches on action output. The selected graph flattens into a transient typed definition and reuses PaceAutomationCompiler plus normal preflight, approval, executor, observation, and audit behavior; no model runs after catalog selection. “Teach a skill …” and “Create an automation …” share a selected-planner authoring ladder: fixed typed automation first, bounded Pace Program second, planner-grounded .skill.md last. The selected model is an authoring aid, never runtime authority.
Teachable skills: .skill.md skills (PaceSkillLoader) are the natural-language sibling of typed automations, Pace Programs, and recorded flows — a numbered step list the planner re-grounds each run (intent tier), rather than compiling canonical tool calls (typed/program tier) or replaying recorded AX steps verbatim (flows, pixel tier). Bundled skills live in Resources/skills/<slug>.skill.md; user skills in ~/Library/Application Support/Pace/skills/. Teachable by telling (PRD docs/product/prds/teachable-skills.md): explicit teach/create transcripts use the shared representation ladder above; Settings → Skills → “Teach a skill” remains an explicit planner-grounded prose form. skillStructuringSystemPrompt + skillFromStructuredJSON use the selected consented planner (local text planner for local tiers), with structureSkillDeterministically as a no-model fallback. Stored triggers are active catalog invocation phrases. PaceSkillLoader.save/deleteUserSkill/listUserSkills use the atomic temp-file+rename pattern and reuse PaceFlowStore.slug(for:). PaceSkillsView surfaces prose skills and deterministic programs in “Your skills.” Saved reusable work stays local; authoring descriptions and saved preferences go to the selected provider when an off-device tier is enabled.
MCP Integration Layer: configured stdio servers use the official Swift MCP SDK with serialized persistent sessions for initialization, live tool schemas, calls, and structured/text results. Configuration changes or failed transports close the session; failed mutations are not replayed. Settings → Connected apps and tools distinguishes installed configuration from an explicit connection test with actual tool count/error. The catalog includes pinned Peekaboo and Playwright helpers and Linear’s official read-only OAuth endpoint through mcp-remote@0.14.3; generic configured tools use their discovered schemas. Missing connections fail visibly. Configuration merges preserve user-added entries. See open-source credits.

Observation, journals, proactivity

Watch mode: PaceScreenWatchModeController is the explicit watch-loop primitive. It samples screenshots, uses PaceScreenImageDiffer, and emits typed events only when a screen has meaningful visual change (majorScreenChange, contentUpdate, focusedRegionChange). The companion panel exposes a Watch Mode toggle, and explicit voice commands such as “watch my screen” / “stop watching” route through PaceWatchModeCommandParser before the planner. Watch events also journal into the local retrieval index (PaceScreenWatchJournal, source screenWatchHistory) so “what did I do today?” questions answer from local history.
Time understanding (journals): two passive retrieval sources power Dayflow-style recall. The screen watch journal records watch-mode events (timestamp, change category, frontmost app, cached VLM description when fresh) into day-per-screen documents; the app usage journal (PaceAppUsageTracker + PaceAppUsageJournal, source appUsageHistory) tracks foreground app minutes and switch counts through permission-free NSWorkspace notifications even when watch mode is off. Both keep 7 days, dedupe/cap entries, rehydrate across restarts, and honor per-source enable/clear in Settings. A third recall source, researchHistory (PaceResearchJournal), captures each research-lane turn as a day-bucketed {question, answer} document (30-day/100-entry retention, same-day dedupe) — recorded by a strictly isResearchTurn-guarded, non-blocking, fire-once hook at the agent-loop turn finalization (question = original transcript, answer = final spoken text) — so “what did I research about X?” answers from history and the turns are browsable in Settings → Memory → Past research.
Proactive features (opt-in): the menu-bar app exposes a small set of proactive surfaces gated by PaceRestraintGate (active-call check, proactive cooldown, intent confidence). Posture watch (PacePostureMonitor), focus-fatigue nudges, calendar pre-meeting nudges, watch-mode observation nudges, and the daily weekday morning brief (PaceMorningTriageScheduler + PaceMorningBriefBuilder) all default OFF and become inert until the user enables them in Settings. When restraint says “stay quiet” the morning brief is parked on a panel card so the user never misses it.
On-device meeting notes: PaceMeetingModeController (in PaceSystemAudioCapture.swift) captures mic + system audio as two separate tracks via PaceMeetingAudioRecorder, segments them into attributed turns via PaceMeetingTurnSegmenter (Accelerate RMS + hysteresis + echo trimming), transcribes each turn on-device via PaceAudioFileTranscriber.transcribeAudioFileSegmented (WhisperKit → Apple Speech fallback), synthesizes structured notes (summary + action items + decisions) via PaceMeetingNotesBuilder using a privacy-pinned LOCAL planner (BuddyPlannerClientFactory.makeLocalOnlyTextPlannerForPrivacyPinnedFeatures() — never the CLI bridge or Direct API tier, regardless of the user’s tier selection), and journals them via PaceMeetingNotesJournal into the meetingNotes retrieval source so “what did we decide in standup?” answers from local history. Fully on-device — no cloud STT, no cloud LLM. The wedge against Granola/Otter/Fireflies (all cloud). Recording files use .part → atomic rename with RIFF-header crash repair. See PRD docs/product/prds/on-device-meeting-notes.md and ADR decisions/0001-meeting-audio-capture.md. Adaptive note profiles (meetily-informed, Pace-native — see openspec/changes/archive/2026-07-12-adaptive-meeting-notes): notes are shaped by a selectable PaceMeetingNoteProfile (bundled Resources/meeting-note-profiles/<slug>.json — general/standup/one-on-one — plus user overrides in ~/Library/Application Support/Pace/meeting-note-profiles/, loaded/validated by PaceMeetingNoteProfileLibrary). The general profile renders byte-for-byte identical to the legacy prompt (compat anchor, default). Selection precedence: explicit selection (voice command or per-meeting panel pick) → non-general default preference → optional local inference (a silent on-device classify call, default OFF) → general. Voice-triggered: “start my one-on-one recording” / “record this standup” starts a meeting with that profile pinned — PaceMeetingModeCommandParser routes the profile name before the recording starts.
Deeplinks and typed-turn scheduling (pace://): external launchers (Raycast, Shortcuts) can trigger Pace via pace://listen, pace://chat?text=..., pace://watch?enabled=true|false, and pace://panel. PaceDeepLinkParser is a pure reject-on-ambiguity parser (500-char chat cap). Listen remains start-only while a turn is active. Typed chat submitted during an active turn enters PaceChatTurnQueue and drains FIFO as each complete turn returns to idle; the panel keeps optimistic user rows in order, shows the pending count, and can clear pending work without cancelling the active turn. Pace deliberately does not run complete turns in parallel because TTS, approval, conversation memory, and Mac actions share one authoritative agent loop and parallel completion would interleave user-visible effects. Each active turn owns a generation lease plus separately tracked routing and response tasks, so explicit cancellation during intent or automation discovery cannot later install stale speech or UI state. Chat/listen turns still run the normal intent/approval pipeline, so a deeplink can do nothing the user’s own voice couldn’t. URL types registered via CFBundleURLTypes; handled in CompanionAppDelegate.application(_:open:) with pre-launch buffering.

Loop, routing, latency

Intent routing: PaceIntentClassifier is a tiny rule-based local classifier. It routes chitchat to a canned response, pure-knowledge to a text-only planner path, and labels screen-read, tool-action, and phone-large-model turns without paying model latency. On eligible Macs, only unmatched .unknown turns use the Apple Foundation Models classifier as a local fallback before the full pipeline.
Plan-act-observe loop: CompanionManager.sendTranscriptToPlannerWithScreenshot runs a multi-step loop. Each step re-screenshots, re-invokes the VLM (heuristic permitting) and the planner, executes grouped tool calls/actions, returns tool observations to the next planner step, and continues until the planner emits [DONE], emits no tool calls/action tags, or hits AgentMaxSteps (default 8).
Speculative planner race (first step only): on a screen-action / screen-description FIRST step, when CompanionManager.speculativeRaceShouldFire passes (race toggle on, local VLM configured, Apple FM available), performFirstStepSpeculativePlannerRace runs PaceSpeculativePlannerRace.raceSpeculative: the in-process Apple FM “lite” planner (transcript + thread-memory only, NO VLM) races the full VLM-fed planner. The full path’s VLM+OCR+AX prep is deferred into the race’s lazy builder so it runs concurrently with lite — lite produces audio in ~150 ms while a cold VLM (2–3 s) is still resolving. Whichever streams first wins TTS; the full path supersedes the lite stream if its first token lands within 500 ms / 60 spoken chars (prepareForSupersedingStreamWithinTurn resets the pipeline so the new stream replays cleanly). Action parsing ALWAYS uses the full planner’s complete text, never lite — the lite path is spoken-feedback only and can’t emit a real action; when lite wins the audio, the lite text becomes the spoken/bubble/memory string so flushFinal can’t double-speak. bothFailed routes to PaceFailureNarrator. Gate false (and every multi-step turn) keeps the single-planner path byte-identical. Toggle: PaceUserPreferencesStore.enableSpeculativePlannerRace (default ON, opt-out in Settings → Planner).
Walking avatar (optional): PaceAvatarOverlay paints a small SwiftUI character at the bottom of the cursor screen in its own tiny NSPanel, but it is not attached at app launch. The default always-visible surface is the menu-bar/notch capsule.

Key architecture decisions

Native Surface Pattern: PaceOnboardingWindowManager hosts a versioned, resumable first-run state machine that migrates the legacy Boolean completion flag, consolidates live permission setup, and reaches one typed or spoken request through submitChatTranscriptFromDeepLink. A truthful response or completed action contracts the in-window signal toward a notch-shaped anchor before the window closes; failure and blocked outcomes do not play success. PaceMenuBarOverlayManager derives the closed Living Notch frame from NSScreen.safeAreaInsets and the two auxiliary top areas and matches the physical housing exactly at idle. Hover and active states add compact horizontal wings at menu-bar height, with state left of the reserved housing width and signal right of it. A deliberate click resizes that same clipped NSPanel equally left, right, and downward, embedding PacePanelChatView inside the surface instead of opening a visually joined second window. MenuBarPanelManager routes show, hide, shortcut, deep-link, and smoke-test requests into the integrated notch only while reliable physical-notch geometry exists. On notchless or external-only display setups, PaceMenuBarOverlayManager creates a standard NSStatusItem, and the same requests open the standalone panel anchored below it. The geometry and fallback choice recalculate after wake and display changes; Pace never draws a synthetic housing. PaceWindowLayering keeps the optional screen-edge activity glow below both the system menu bar and Living Notch while deliberate teaching annotations remain topmost; the glow defaults off and is transparent while idle. Delayed onboarding transitions are cancellable and generation-guarded so stale work cannot advance a newer scene.

Command Center Pattern: PaceMainWindowManager owns one normal resizable NSWindow for every durable management surface. PaceCommandCenterRouter groups primary destinations under Use Pace, Activity & Privacy, Customize, and Help, while optional integrations and developer controls use one searchable Advanced disclosure. It honors exact requested routes and restores the last valid selection. PaceSettingsWindowManager is a compatibility facade into the same window rather than a second host. The notch panel remains a present-turn surface; its gear routes to Customize → General. Pace still runs as LSUIElement=true, and the Command Center explicitly activates the app only when opened.

Cursor Overlay: A full-screen transparent NSPanel hosts the blue cursor companion. It’s non-activating, joins all Spaces, and never steals focus. Cursor flight, response text placement, and pointing animations render in this overlay via SwiftUI through NSHostingView; listening/thinking animation lives in the notch bar.

Global Push-To-Talk Shortcut: Background push-to-talk uses a listen-only CGEvent tap instead of an AppKit global monitor so modifier-based shortcuts like ctrl + option are detected more reliably while the app is running in the background.

Transient Cursor Mode: When “Show Pace” is off, pressing the hotkey fades in the cursor overlay for the duration of the interaction (recording → response → TTS → optional pointing), then fades it out automatically after 1 second of inactivity.

Local-mode setup

See SETUP_LOCAL.md for the full recipe. The complete Info.plist switch reference (local VLM / planner / TTS-sidecar / transcription-provider knobs and their defaults) lives in ../development/info-plist-switches.md.

Everyday desktop actions and Codex sessions

The selected consented planner can compose app launches, browser/profile opens, screenshot controls, screen-recording controls, meeting-audio controls, and codex_session calls. These are general tools, not bundled work-startup presets. Chrome’s saved work-profile directory is used for profile: "work" and ordinary Chrome opens. Native screen capture still requires the user to choose a region and press Capture/Record. Each launch preserves the native destination settings, selects the requested mode through the native toolbar using Accessibility, and confirms its Capture/Record label before reporting success. Immediate launch failures are surfaced. start_recording, stop_recording, and recording_status use PaceScreenRecordingController to record the main display with the native screencapture utility, save locally under Pace/screen-recordings, and validate the movie’s duration and video track before reporting a successful stop. It stops only its owned process; shutdown requests normal recording completion. Meeting observations report the recorder’s actual state.

codex_session validates an existing absolute or tilde-expanded directory, resolves the installed Codex executable, writes a uniquely named Warp Tab Config, and opens it through Warp’s URL scheme. The folder is data, not shell code; Codex’s interactive sign-in, folder trust, and action approvals remain intact. Warp 2026 Tab Config support is required. Pace reports launch requested rather than claiming it inspected the terminal session.

Foreground CLI-direct planning works immediately after explicit consent. The 24-hour first-use gate remains for unattended scheduled tasks. CLI planning runs in a private temporary directory, independent of Git, with user config, shell, hooks, apps, browser, computer-use, and multi-agent tools disabled. Research explicitly enables web search. The stream runner handles UTF-8 boundaries, drains stderr to a private file, cancels the subprocess, and bounds calls with a timeout. CLI turns avoid local VLM prewarming and use permitted images plus AX/OCR context. Research and Read My Screen off skip automatic screen capture entirely.

Lightweight computer use and text dogfood

Peekaboo 4.9.0 is the optional OSS Mac automation backend, installed through the existing MCP catalog with a pinned npm package. It needs Node 22+ and its own OS permissions; Pace does not provision another AI provider. The catalog explicitly starts mcp serve --allow-foreground so app launches and foreground UI work, with that capability disclosed on the install card and individual actions still gated by Pace approval. Existing background-only installs need removal and reinstallation from the catalog; Pace does not silently expand their configuration. Pace discovers its actual tool schemas and supplies them to the selected planner. Peekaboo observations/actions reuse one serialized MCP process because its snapshot IDs belong to that producer; changing configuration or a transport failure closes the session. Failed mutations are never replayed automatically. App exit closes owned persistent sessions. Read My Screen gates Peekaboo window inspection/control, while read-only app inventory remains available.

Text commands use the same task routing and typed execution pipeline as final voice transcripts. Implicit reusable-work execution currently requires an exact authored invocation phrase. Semantic similarity and local-model ambiguity resolution no longer authorize a saved routine; uncertain requests fall through to the selected planner. Explicit automation/catalog commands still work. Native screenshot commands open macOS controls. Explicit start/stop/status screen-recording commands use the owned recording lifecycle; opening selection controls remains a separate action.

Playwright MCP 0.0.83 supplies browser navigation, snapshots, and element actions in a separate isolated Chrome session. Its persistent producer preserves browser state across tool calls. Both Playwright and Peekaboo use the official Swift MCP SDK, pinned to revision a0ae212ebf6eab5f754c3129608bc5557637e605 for its upstream transport concurrency fix on the current Swift compiler. All configured stdio servers use that SDK transport and preserve producer state across calls.

Local speech uses WhisperKit base.en from Application Support, provisioned with python3 scripts/setup-whisperkit.py; pinned model and tokenizer assets avoid recognition-time downloads. This small default model recognises English. Bundled Silero v6 CoreML rejects silence before streaming/file decoding. Explicit read aloud using Kokoro ... starts the existing mlx-audio sidecar on demand and uses mlx-community/Kokoro-82M-bf16; ordinary voice selection stays configurable. transcribe audio <path> and check for updates reach the existing transcription and Sparkle components through the shared text/voice router. Meeting capture requests microphone access asynchronously before opening the audio engine to keep the permission prompt off the blocking device-binding path.

Native Vision OCR performs synchronous work on a detached task rather than the UI actor. CLI image planners receive screenshots plus AX context directly, so their screen turns do not wait for the local OCR/VLM path. A stop-meeting text command bypasses the turn queue while capture startup is awaiting permission.

Durable read-only work and recurring assistance

Background summaries, drafts, and research persist task state and full results atomically in Application Support. Queued/running work becomes interrupted on restart and requires explicit retry from the beginning. Cancellation guards late results by execution ID. Scheduled origin survives persistence and retry; its off-device consent/24-hour gate is rechecked at execution time.

Recurring tasks preserve exact intervals or local daily times, persist the next deadline, coalesce missed intervals into one run, and prevent overlapping runs. Pause, deletion, and global disable are rechecked after asynchronous preflight. Interrupted claims require review. The Tasks surface exposes next run, errors, results, cancellation, and explicit retry. Morning briefs retain their readable card and delivered day before optional speech, with same-day catch-up on launch or wake. These background paths generate read-only results, never execute foreground mutations silently.

Requested Files, Notes, Mail, and Calendar summaries refresh their permitted sources before answering and include provenance. Disabled sources are filtered both during unified-memory sync and recall. Explicit approved file roots support PDF/RTF/text with bounded enumeration fallback when Spotlight has no index; secret paths and symlink escapes remain excluded. Missing permissions, empty accounts, and unavailable sources are reported rather than replaced by stale personal facts. Notes/Mail refresh uses only existing AppleEvents grants.