Why we read the screen with OCR instead of trusting selectors
Full-frame OCR on our busiest screen, a map UI with tabs and a legend, recovered two lines of text. The same screenshot cut into a 3-by-3 grid of overlapping tiles, OCR'd per tile and merged by overlap, recovered 34 lines including every tab label. It was also faster: 0.53 seconds against 2.18. That one measurement shaped how our operator reads the screen, and why it is never allowed to read your numbers.
The measured envelope
We benchmarked Apple's on-device Vision OCR against DOM-visible text as ground truth across seven real pages. Accurate mode: word recall 86 to 100 percent, mean 94, at 270 to 620 ms per screen. Fast mode: recall 27 to 95 percent, mean 77, at about 127 ms.
That 27 is not a typo, and it's the number that matters most. Fast mode will read a page plausibly and drop the one field you cared about, with no error. So fast mode gets exactly one job, classifying what kind of page we're looking at, and is banned from reading values. Precision floors at 71 to 93 percent, partly because text baked into images and SVG counts against the OCR as spurious even when the read is right.
The rule: numbers come from the DOM
OCR decides where to click. It never supplies a value we store or assert on. A parcel number, a price, a count is read from DOM text or a network response, full stop. A 94 percent mean recall is a fine clicking accuracy and a terrible ledger. This is the line most "AI watches the screen" pitches blur, and blurring it is how a test suite starts asserting on hallucinations.
Where reading the screen wins
Selectors stay the default, and generated tests emit real Playwright locators. The operator exists for pages where the DOM is hostile: document portals whose markup churns every release (the Laserfiche and EnerGov class of government software), canvas UIs where the button is a region of pixels, embeds you don't control, apps mid-redesign.
The loop: screenshot, tiled OCR, a cheap text model picks which visible label serves the goal (roughly $0.0002 per decision, with the model id and cost logged on every step), a curved and jittered mouse path, click, re-OCR to confirm the result. First real outings: three map tabs clicked purely by their OCR'd labels, confirmed 3 of 3 by re-reading the screen, and a state health-department records portal where "open the permit PDF" took the model exactly one decision.
Reading the screen is no longer only a runtime fallback, either. The agent designs OCR-anchored flows: ocr-click and ocr-type steps that target visible text instead of a locator, saved and replayed like any other flow. On a canvas UI or a no-DOM embed there is no selector for a step to hold, so the OCR anchor is the step, not a patch over one.
Where it silently fails
The failure mode to respect is not "no text found." It's a confident wrong read: a word split across a tile boundary becoming two wrong words when the merge step is missing, fast mode dropping digits, an instrument that cannot see what it's pointed at reporting success anyway. Our standing rule for any detector, OCR included: show it a known positive before you trust its zero.
Selectors get the last word
Where the DOM supports it, codegen emits locators once a flow works, and the test never pays the OCR or model cost again. Where it doesn't, the flow keeps its OCR anchors, because visible text is the only stable thing on that page. Off macOS the same interface is served by a vision model instead of the Vision framework, so the approach degrades rather than dies. Vision stays the option of last resort. The difference here is that our last resort ships with an error bar.
Related: PASS / FAIL / UNMEASURED: why a QA verdict needs a third state · How mapping sessions work · Compare