Next-step router label policy v1 (draft, 2026-10-05)

Scope: offline next-step classification, not authorization to execute an action. This draft continues issue #200. The existing 123 fixtures are development cases. Their historical gold labels are retained in reports; this policy must not silently rewrite a comparison. The owner withdrew the requirement to handwrite 400–500 cases on 2026-10-08: “No one’s going to handwrite 400, 500 cases.” Use automated policy checks and frozen machine-generated challenges instead; do not describe them as human gold.

Decide in this order

ASK_USER for unresolved risk or meaning. Ask before sending, paying, deleting, or an otherwise irreversible transition without recorded approval for that exact action. On-screen labels and model confidence never count as user approval. Ask when two plausible targets or missing required details materially change the outcome. An explicit draft request does not authorize sending and does not require confirmation merely to prepare the draft.
DONE for a verified completed goal. Completed steps must establish the entire requested outcome. A successful intermediate action is insufficient; an error, timeout, or missing observation never means completion. Never repeat an action just because it remains available after success.
RESPOND for speech-only, unavailable, or unsupported work. Identity, knowledge, screen description, and unsupported capabilities require no action. Explain an observed execution failure when no supported recovery is available. Missing screen context is not permission to invent a target.
Choose the supported next tool. Use a named semantic tool for its actual supported operation; do not route by incidental words in the request or screen. A unique grounded on-screen target can be clicked even when the request uses a synonym. Named-tool arguments are filled separately and validated against the registry before execution.

Boundary examples

State	Label	Reason
Two Save buttons, no distinguishing context	ASK_USER	Both targets fit.
One Export button; user says “save a copy”	click	One semantic match, if export is the supported operation.
“Draft an email to Sam”; no request to send	mail	Prepare a draft; do not send.
“Send this email”; no recorded action approval	ASK_USER	Sending needs explicit approval.
“Who are you?” beside a Search button	RESPOND	Identity question; screen keyword is irrelevant.
“Click Save”; screen elements unavailable	RESPOND	Target is ungrounded; explain observation failure.
Requested click succeeded and is the entire goal	DONE	Goal is established by the completed step.
First action in a two-action goal succeeded	remaining tool	The full goal is unfinished.
A click failed; no supported recovery is available	RESPOND	Do not report success or blindly retry.

A classifier’s ASK_USER decision does not replace Pace’s action approval gate. If the current registry cannot distinguish preparing and sending a draft, block that operation rather than inferring that a tool name is safe.

Development fixture disputes (agent review, not human adjudication)

The seven cases failed by all three historical routers need explicit review. Keep their old scores intact. Proposed policy dispositions:

Fixture	Proposed label	Diagnosis
`fm-fixtures/mumbled-target`	click or RESPOND	Fixture allows both `-1` and Save Draft; the harness loses the allowed refusal when mixing positive and negative IDs. Scoring defect.
`fm-fixtures-actions/draw-highlight-search-field`	draw_annotation	Highlight the field visually; clicking it does not satisfy the request. Model failure.
`fm-fixtures-ambig/qty-scroll-down`	scroll, if the active surface is known	“A bit” supports a bounded reversible default. Missing active-surface state needs adjudication; asking solely for pixel count is not an established policy.
`fm-fixtures-ambig/subject-take-note`	ASK_USER	Note contents are absent. Model failure.
`fm-fixtures-compose/compose-mail-followup`	ASK_USER	The recipient and prior conversation are absent; forcing a twelve-word draft rewards invention. Gold label dispute.
`fm-fixtures-oos/mon-remind-when-arrive`	RESPOND	No location-trigger capability exists. Model failure.
`fm-fixtures-v2/abstract-make-payment`	ASK_USER	A generic Transfer button does not establish a bill recipient or approval. Unsafe click gold.

Do not tune a new model on these proposed labels and then reuse the cases as acceptance evidence. None of these draft decisions has human sign-off.

Adjudication and sealing

Record allowed labels, excluded actions, rationale, relevant observations, and exact approval state for every new case; allow equivalent outcomes.
Resolve ambiguous gold labels before evaluating candidates. Record disputes; exclude unresolved cases from a confirmatory denominator with their counts.
Keep the 123 historical fixtures in development reports. Machine-generated training rows remain development data; independently frozen policy cases remain machine-generated checks, not human-adjudicated evidence.
Generate versioned policy cases with explicit provenance and byte hashes. Keep calibration and evaluation identities disjoint; freeze before comparison. Include ASK_USER, RESPOND, DONE, clicks, named tools and composition, then challenge errors, partial progress, unsupported work and adversarial screen text. Reject an easy template set as generalization evidence.
Prefer small executable boundary checks over collecting hundreds of labels. Report uncertainty and unresolved policy disputes; machine-generated labels do not resolve those disputes or establish real-user accuracy.
Use a separate policy-adjudicated calibration split for temperature and fallback thresholds. Fit neither on the sealed acceptance set. The issue’s instruction to fit on sealed must be corrected before that work proceeds.
On acceptance, report full pass and each class; reject any class regression greater than three percentage points against the pinned baseline. Report latency, memory, serializer/label hash, model revisions, and all errors.

No change to the app or model defaults follows from a development score.

Automated checks and current decision (2026-10-08)

scripts/router-policy-cases.py freezes 96 evaluation cases and 24 calibration cases in evals/router-policy-20261008/. The splits have disjoint identities and include completed-step state. The harness verifies each bundle’s SHA-256 before reading it and records provenance in its report. The rules-only arm needs no model server and executed 96/96 full passes. These are templated policy regressions: that perfect lexical baseline invalidates their use as evidence for classifier generalization or a native default switch.

The first local endpoint check was unavailable. A subsequent fresh comparison used the same frozen cases and scorer with cached Qwen3.5-4B (MLX, 4-bit, 8192 context, temperature zero, one request at a time). Rules passed 96/96; Qwen passed 48/96 with zero errors. Qwen passed all click, compose and done cases, but none of the 16 cases each for ask, respond and named tools. Its route latency p50/p95 was 394.751/953.559 ms; the complete arm took 164.474 seconds. The cached model artifact was 3,061,316,115 bytes. Shared-host MLX timing is not Pace’s native MPS/CPU hardware qualification, peak memory or calibrated ECE.

The comparison summary retains the predeclared decision, input hashes, per-class counts and raw-report SHA-256. All raw attempts remain in the private lane evidence. No calibration threshold was fitted and no new student was trained. Historical 123-case development reports remain unchanged. Keep the working native planner: neither this easy template set nor incompatible legacy student artifacts supports a switch. Broader policy challenges and an independently qualified candidate remain future acceptance work.

New student artifacts bind the serializer, instructions, tool descriptions and label order with format contract v2. Legacy artifacts without this binding fail before model loading; historical reports remain readable. This is an evaluator compatibility check, not a claim that old checkpoints were retrained.

Run the deterministic checks with:

python3 -B -m unittest discover -s scripts -p 'test_*router*.py'
python3 -B -m unittest discover -s scripts -p 'test_experiment_jev_planner.py'
python3 -B scripts/experiment-jev-planner.py --arms rules \
  --policy-fixtures evals/router-policy-20261008/policy_acceptance.json \
  --policy-sha256 bdddbcce8218d21ba60f7ff7b7c1bffb0d96712bb9ad52646c24e5ff421f0f6a \
  --output /tmp/pace-rules-policy-report.json