Skip to content
All skills

Use AgentVision

See and interact with the user's screen using screenshots, element discovery, clicks, form filling, and visual QA

v1.2.0SkillRobin van Baalen/use-agentvision:use-agentvisionSource
/plugin install use-agentvision@rvanbaalen

When to use

Use when Claude needs to see and drive what is on your real macOS screen: UI development feedback loops (before/after screenshots), visual QA of running apps, filling forms, navigating desktop applications, and testing iOS Simulator or Android emulators. Not for headless browser testing or Playwright/Puppeteer scripts.

Requires the agent-vision CLI (brew install rvanbaalen/tap/agent-vision) plus Screen Recording and Accessibility permissions.

How it works

  1. Opens a session with agent-vision open <App>, which launches or activates the app and selects its window automatically — or agent-vision start for a manual region drag
  2. Scans the region with agent-vision elements to build an indexed list of accessibility and OCR elements
  3. Acts on a discovered element — click, type, key, scroll, drag
  4. Waits for the UI to settle, then re-scans, because element indices go stale after every update
  5. Captures screenshots and reads them back to verify outcomes visually
  6. Stops the session once you confirm you’re done

Capabilities

Session management — every command is scoped to a session UUID covering a selected screen region. open handles app launch and window selection; start covers custom regions.

Element discovery — indexed scans over the Accessibility API and OCR. Complex apps return 80+ elements, so scans are filtered with jq down to interactive accessibility nodes.

Focus-free targeting — --element N drives the Accessibility API without moving the cursor or stealing focus, so you can keep working in another window. --at X,Y is the fallback for canvas and custom-drawn UI, and is always verified with agent-vision preview first.

Screen capture — screenshots of the region on demand, read back and described for before/after diffing and QA reports.

File sharing — attachments go in via an osascript clipboard copy plus Cmd+V, not by driving Finder dialogs.

Background delegation — long multi-step flows are dispatched to a background sonnet subagent that echoes one-line progress markers, streamed back through the Monitor tool while the main thread stays free.

Ground rules — never resizes or rearranges windows, never works outside the selected area, never sends blind keystrokes, and verifies outcomes on screen rather than through shell commands.

Invoke

/use-agentvision:use-agentvision

The skill also activates automatically when you say things like “look at my screen”, “take a screenshot”, “fill this form”, “check the UI”, or “I have X open”.