Technique

Making an API out of a 2004 web portal

2026-08-29 · The QA Reef team · 4 min read

We pulled 14,753 permit records out of a Washington county's Laserfiche WebLink portal. WebLink is 2004-vintage ASP.NET, from the .aspx-and-__VIEWSTATE era of the web: postback forms, session cookies, a document viewer that wants you to click "Download & Print" inside a session it controls. No API, no docs. This is how it became an endpoint anyway, and what actually broke on the way.

The browser is a debugger, not a runtime

Version one drove the viewer with Playwright: open DocView.aspx?id=N, click through to the PDF, catch the download. It worked, at roughly 15 seconds per document with a warm Chromium per portal. Fine for a demo. At 14,753 documents that's two and a half days of a browser doing what three HTTP calls do.

Because that is what the "Download & Print" button turns out to be. Watching the wire during a replay: POST GeneratePDF10.aspx returns an export GUID, POST DocumentService.aspx/PDFTransition polls until the export finishes, GET /PDF10/<guid>/<id> is the PDF. Folder listings are the same story (FolderListingService.aspx/GetFolderListing2, a JSON POST). The entire portal is a thin skin over about six requests.

We later put "how do you crack a JS-gated records portal" to two frontier models as a design review, and both independently flagged the same move as, in one's words, often the whole crack: the page fires a background XHR whose response IS the document, so grab that response instead of driving the DOM to a viewer. That rule is now a module in our stack (netcapture): during any replay it watches every response, keeps anything that sniffs as a real artifact (starts with %PDF, big enough not to be an error stub), and refuses HTML served under a PDF-shaped URL, the classic error page wearing a document's clothes. The exchange is also kept as a HAR, so the requests behind a working replay can be read back and re-issued without the browser.

Three things that actually broke

Pagination lies. GetFolderListing2 with getNewListing:true resets the server-side cursor to page one, caps the response at 40 entries, and silently ignores your start offset. Only getNewListing:false on every later page continues the cursor. Until we caught it, every folder appeared to hold exactly 40 entries. Verified live, then regression-tested, because a bug found by staring comes back when only your memory guards it.

Arrays lie too. Self-hosted WebLink pads a small folder's results out to a fixed page size with nulls. A naive .map() over the raw array throws. .filter(Boolean) first.

Sessions wedge. Call GeneratePDF10 without a warmed session and the server answers "Session lost." Lose the session mid-run and JSON endpoints start answering with HTML. Our harvester re-warms exactly once and retries exactly once, never loops. And the export route only declares itself dead after 25 consecutive failures with zero documents saved that run: a few flaky exports on a portal that is otherwise yielding must not abort a 25,000-document job, and a truly dead route must not spin forever.

The payoff

The production harvester is now plain Node: global fetch, a hand-rolled cookie jar, zero page.evaluate. The day we migrated three of these jobs off browsers, the Chromium process count on our runner box dropped from 82 to 40, and the swap pressure that had been killing browsers mid-call went with it. The same code runs against Laserfiche Cloud and self-hosted WebLink by swapping a base URL, because underneath it's the same six requests on a different path.

Once a flow is a function, the endpoint is the easy part: each portal is a small class, and a thin HTTP facade exposes every public method as GET /<site>/<method>, one warm session per site, calls serialized so two requests never fight over one session.

Where the browser still earns its seat

First contact. Finding the repository, reading a screen of decades-old folder names, learning that the records live under "Community Development / Building," warming a session the HTTP layer can then reuse: that's recorder and OCR work. The browser explores; HTTP executes. And the hard line stays where it always is: no defeating a CAPTCHA or an auth wall on a portal you aren't authorized to query. A stop gets recorded as a stop. More on that boundary: legacy system automation and turning a website into an API.