Technique

Making an API out of a 2004 web portal

2026-08-29 · The QA Reef team · 8 min read

A lot of the software that still runs the world was built before "API" meant anything to whoever built it. County record systems, permit portals, old case-management tools — frame-based layouts, form posts that reload the whole page, result rows with no consistent markup between two searches on the same site. You can't curl your way into one of these. You also can't record a Playwright selector against it and expect that selector to survive the next search. This post is about the actual technique we use to turn a portal like that into something callable — not a sales pitch, the mechanics.

Start from what a person actually sees

The core move is refusing to reason about the DOM at all for the parts of the page that don't have one worth trusting. Instead: take a screenshot, run OCR on it, and work from words with pixel positions — the same information a person has when they look at the screen. This inverts the usual automation assumption. A selector-first tool asks "what element has this id/role/text," which presupposes the markup is stable and meaningful. An OCR-first approach asks "what text is visible, and where" — a question that has an answer even when the markup underneath is a table of nested tables with no semantic structure at all, which describes a startling amount of pre-2010 government software still in production.

Tiling matters more than the OCR engine

A single full-page OCR pass on a dense results table or a form-heavy search screen loses text — small labels near table borders, text at the edges of the viewport, anything crowded. The fix isn't a better OCR model, it's tiling: cut the screenshot into overlapping crops, OCR each tile independently, then merge overlapping detections by IoU (intersection over union) so a word that got split across two tile boundaries doesn't turn into two separate, wrong detections. In our own internal testing against a DOM ground truth on dense, map-heavy screens, tiling was the difference between recovering a couple lines of text and recovering three dozen — the underlying OCR engine (Apple's on-device Vision framework, on macOS) didn't change, only whether it got a fair look at the whole screen. Measured word recall across a range of busy screens landed between 86% and 100%, with a 94% mean.

From words to a click: coordinates, not selectors

Once OCR returns a list of words with bounding boxes, the next step is deciding which one to act on. For a scripted flow (you already know you want "Search" clicked), that's a direct lookup: find the box whose text matches, take its center point. For an agent-driven flow (a goal like "find the record for parcel 04-1234-000"), a cheap text model gets the OCR'd word list and the goal, and picks which visible label to act on next — a narrow decision, not a model reasoning over raw pixels from scratch, which keeps this cheap enough to run per step.

Either way, the output is a pixel coordinate, not a selector. That's a deliberate trade: a coordinate click works on literally anything rendered on screen — a canvas, a plugin-rendered PDF viewer, a table with no id attributes anywhere — at the cost of being less semantically meaningful than page.getByRole('button', { name: 'Search' }). For a portal with no stable DOM to begin with, that trade is free; there was no selector to lose.

The mouse moves like a person's, on purpose

The click itself doesn't teleport the cursor to the target and fire a synthetic event. It moves along a curved, jittered path from wherever the cursor currently is to the target point, the same rough shape a human hand traces with a mouse — not perfectly straight, not perfectly smooth. This matters for two independent reasons. First, some of these older systems have JavaScript wired to mousemove events (hover states, tooltips, occasionally naive bot-detection) that a teleporting synthetic click never triggers, silently breaking the flow. Second, and just as important for evidence: a recorded path is something you can actually review afterward and confirm "yes, this went where it looks like it went," rather than trusting a click event fired correctly against a target you can't independently verify landed.

Evidence per step, not just a final result

Every action — not just the final one — keeps a screenshot, the OCR'd text it acted on, and (for agent-decided steps) the exact model that made the call and what it cost. This isn't optional logging bolted on afterward; it's the actual point of automating something you don't control. When a county rolls out a portal redesign overnight and a previously-working flow starts failing, the evidence trail is what tells you whether the search form moved, the results format changed, or the site is just slow today — without re-running anything blind and guessing.

What happens when the page never settles

This is the case that separates a demo from something you'd actually rely on. Old portals do things modern SPAs mostly don't: a search that sometimes takes eleven seconds and sometimes forty, a results page that partially renders and then silently stops, a session that times out mid-flow with no visible error. A naive automation either times out and reports a hard failure (wrong — the target might be completely healthy and just slow) or waits forever (worse). The honest answer is a third state: if a check never gets a clean, stable look at what it's testing — the OCR pass keeps returning a half-rendered page, or the same "loading" text three tiles in a row — it reports UNMEASURED, with the evidence it collected attached, instead of forcing that ambiguity into a false PASS or a false FAIL. A monitoring system built on top of that distinction can alert on a real pattern of UNMEASUREDs (the portal's gotten slower, worth investigating) without treating every timeout as "the site changed and broke."

Turning the working flow into an endpoint

Once a flow reliably does what it's supposed to — search, read results, fetch a document — wrapping it as an HTTP endpoint is the easy part. Every site in the codebase is a small class with public methods; a thin, dependency-free HTTP facade (src/api.js in the QA Reef repo) turns each public method into a route automatically: GET /<site>/<method>?arg=value, or a POST with a JSON body, with one browser kept warm per site and calls to the same site serialized through a queue so two requests don't collide on one page. The interesting engineering was making the flow itself reliable against a portal that fights you at every step — the endpoint wrapper is a few dozen lines on top of that.

What this doesn't do

None of this touches a CAPTCHA, an auth wall, or anything designed to stop automated access — those get recorded and reported as a stop, not defeated. And none of it is a substitute for actually knowing you're allowed to be there: this technique is for systems you're authorized to operate, whether that's your own internal tool or a public records portal you have a legitimate reason to query, not a general-purpose way around access control. More on where we draw that line: legacy system automation and turning a website into an API.