Target Resolution

Overview

Target resolution converts fuzzy natural language references into concrete, actionable targets that the execution pipeline can operate on. “Open the email from John” does not specify which email client, which message, or which John. The resolver’s job is to eliminate that ambiguity and produce a resolved target: an application ID, surface ID, and element ID that every downstream stage can use without further interpretation.

The resolver runs through six strategies in priority order, starting with the fastest and most precise and falling back to slower, more contextual approaches. If every strategy exhausts its candidates without a unique match, the resolver returns a structured error that lets SI ask the user for clarification.

graph TD
    A["Incoming Intent"] --> B{"Exact ID match?"}
    B -->|Yes| Z["Resolved"]
    B -->|No| C{"Role + surface title?"}
    C -->|Yes| Z
    C -->|No| D{"Text content search?"}
    D -->|Yes| Z
    D -->|No| E{"Fuzzy text match?"}
    E -->|Yes| Z
    E -->|No| F{"Contextual disambiguation?"}
    F -->|Yes| Z
    F -->|No| G{"Multi-candidate prompt"}
    G --> Z

Resolution Strategies

1. Exact ID Match

The fastest path. If the intent contains a direct element ID (such as atspi:47-w1-btn-compose), the resolver looks it up in the element cache and returns immediately. No scoring, no heuristics. This path handles programmatic references and SI’s own internal re-invocations.

Latency target: under 1ms.

2. Role + Surface Title

Matches by combining a semantic role (button, text field, menu item) with a surface title. For example, “the send button in the Mail window” matches an element with role push-button, label “Send”, on a surface titled “Mail”. The resolver checks AT-SPI2 role attributes and surface metadata from the compositor.

Latency target: under 5ms.

3. Text Content Search

Searches element labels, names, and accessible descriptions for a string match. This catches references like “the compose button” or “the search bar” where the role is implied but not stated. The search runs against the element cache, not the live accessibility tree, so it avoids D-Bus roundtrips.

Latency target: under 10ms.

4. Fuzzy Text Match

When exact text search fails, the resolver applies fuzzy matching (edit distance, substring matching) against element labels. This handles misspellings, partial names, and paraphrased references. Candidates with a similarity score above a configurable threshold advance to the next stage.

Latency target: under 15ms.

5. Contextual Disambiguation

Uses session context to rank candidates: focus history, recent interactions, spatial position within the viewport, and the application’s current state. If the user was just interacting with a specific window, elements on that surface receive a higher ranking. This strategy resolves most real-world ambiguity without user involvement.

Latency target: under 20ms.

6. Multi-Candidate Clarification

When multiple candidates remain and contextual disambiguation cannot break the tie, SI prompts the user to choose. This is the slowest path because it blocks on user input. The prompt presents ranked candidates and the user’s selection is recorded for future resolution.

Latency target: under 2s (user-response-bound).

Default App Resolution

When multiple applications share a capability (two browsers, two email clients), the resolver needs a deterministic way to pick one. It evaluates a priority chain in sequence:

1. User-configured default. An explicit preference the user set. Checked first.
2. Contextual relevance. The app is already running and focused. If so, use it.
3. Last-used. The most recently used app for that capability. Decays over 30 minutes to prevent stale preferences.
4. Capability tier score. Higher-tier apps are preferred. Native (Tier 1) ranks above AT-SPI2 (Tier 2), which ranks above structural (Tier 2), which ranks above minimal (Tier 3).
5. Fallback. Ask the user. The answer is remembered as the new default.

This chain ensures that repeated requests resolve consistently unless the user’s context genuinely changes. The last-used decay prevents a single interaction from permanently overriding a deliberate default.

Edge Cases

Scenario	Behavior
No matches	Returns TargetError::NotFound with a contextual hint suggesting what might have been intended
Multiple matches, same surface	Prefers the focused element, then the most recently interacted element, then the element closest to viewport center
Multiple matches, different surfaces	Prefers the focused surface, then the last-used surface, then prompts the user
Element not accessible	Returns TargetError::NotAccessible
App not running	Returns TargetError::AppNotRunning with an option to launch the app

Cross-App Coordination

V4.2 introduces three coordination primitives for composing actions across multiple applications. These are admin-only operations:

• coordinate.sequence.execute runs a series of actions in order, stopping on the first failure (fail-fast serial).
• coordinate.parallel.execute runs multiple actions simultaneously, waiting for all to complete (gather-all).
• coordinate.conditional.execute dispatches on a condition: if the value is non-null, non-false, non-zero, or non-empty, the “then” branch runs; otherwise the “else” branch runs.

Each coordination operation publishes a coordination.executed event and is recorded in the coordination history log.