Verification

PASS / FAIL / UNMEASURED: why a QA verdict needs a third state

2026-08-29 · The QA Reef team · 4 min read

For three hours, our checks said a production fix was live. Two separate scripts fetched the page and confirmed the new bytes. Users kept getting the old page. The scripts used curl, which defaults to Accept-Encoding: identity; browsers ask for gzip, and nginx's gzip_static was serving a stale index.html.gz to every real browser while the uncompressed file was fresh. Both checks agreed because both had the same blind spot.

Nothing in that story was a PASS, and nothing was a FAIL. The checks never observed what a user sees, so they had no verdict to give. Most test systems have nowhere to put that fact, and that is the bug this post is about: a measurement you can't trust must never be allowed to impersonate a verdict, in either direction.

"Renders" is not "live"

The gzip incident is one instance of a shape we now check for by name. A harness that pre-picks application state renders a perfect page while bypassing the gate a real user has to clear. A check that re-verifies its own fixture confirms the fixture. Two checks with the same blind spot count as one check. So: verify with the request shape users actually make, on a fresh page, through the front door, or record the result as unverified. Not failed. Unverified.

Measure the box before blaming the remote

One night, two recurring errors, "browser has been closed" and a storage upload failure, turned out to have three causes between them, and none was the remote service. The disk was at 100 percent, which killed the upload helper, which surfaced as a failure with an empty error string. Two overlapping worker processes shared one log file, so a fatal line from the dead one read as a live failure. And swap was exhausted: 12.1 of 13.3 GB used, 82 Chromium processes on a 16 GB machine, the OS reaping browsers mid-call.

The rule we wrote down afterward: an empty error string means your local helper died, not that the remote said no. Check the disk, the swap, and the process count before filing a bug against someone else's server. Each of those conditions would otherwise have produced a confident-looking FAIL, someone paged, an hour spent bisecting commits for a regression that never existed.

UNMEASURED is a verdict

So our runs report three states, judged against criteria derived from the prompt that defined the check. UNMEASURED is not a maybe between pass and fail. It is a distinct claim: this check did not get a valid observation of the thing it was supposed to verify. A viewport that drifted, a source that never finished loading, a self-test that failed before the real check started: all UNMEASURED.

Two hard rules make the state worth having. It never collapses into FAIL. And it never counts as a zero when results roll up: "6 failed, 3 unmeasured, 1 passed" and "9 failed" are two different investigations, and flattening the first into the second hides exactly the information you needed.

Where it bites in practice

Concrete places the third state does work in QA Reef. A self-heal that fell all the way back to clicking recorded coordinates reports UNMEASURED, not PASS, unless the target is a canvas, because "the mouse went somewhere" is not "the right control was clicked." A run where more than a fifth of steps needed healing is UNMEASURED with the message "the app changed materially, re-record," because a run held together by that many patches is no longer measuring the app. And bugs are auto-filed from FAIL only, never from UNMEASURED, so the bug tracker records failures of the app, not moods of the harness.

What it buys

The cost is real: sometimes you don't get a clean answer, and a three-state dashboard takes one more sentence to explain. What you buy is a FAIL people believe. Once red reliably means "the check got a clean look and what it saw was wrong," teams investigate red. Suites without the third state train their teams to route around red instead, and that is how a suite stops meaning anything, quietly, long before anyone notices.