PASS / FAIL / UNMEASURED: why a QA verdict needs a third state
Most test runners have two outcomes. A check runs, and it either passes or it fails. That's the right model when the check actually got a clean look at the thing it's testing. It's the wrong model the moment it didn't — and "didn't get a clean look" happens constantly in browser automation: a page that hasn't finished loading when the assertion runs, a viewport that scrolled somewhere unexpected, a third-party script that hung, a network hop that timed out for reasons that have nothing to do with your app.
When a binary system hits one of those cases, it has exactly two places to put the result, and both are wrong. Call it a pass, and you've asserted something you never actually observed. Call it a fail — the more common choice, since most frameworks default an exception or a timeout to red — and you've told a human "your app is broken" when the honest statement is "I don't know." Those aren't the same claim, and treating them as the same claim is where false alarms and, worse, false confidence both come from.
What UNMEASURED actually means
QA Reef's runs, and the checks its harness code runs internally, report one of three verdicts: PASS, FAIL, or UNMEASURED. UNMEASURED isn't a hedge or a "maybe" between pass and fail on some sliding scale — it's a distinct claim: the check did not get a valid observation of the thing it was supposed to verify. A drifted viewport, a source that never finished loading, a self-test that failed before the real check even started — all of these are UNMEASURED, not FAIL, and the rule that makes the state useful is strict: UNMEASURED must never silently collapse into FAIL, and it must never be counted as a zero when you're aggregating results (rolling ten runs up into "9 failed" when really "6 failed, 3 were never measured, 1 passed" hides exactly the information you needed to fix the right thing).
The failure mode this prevents
The reason this is a first-class state rather than a nice-to-have comes from a specific, repeatable shape of bug: a measurement is contaminated by the harness itself, not the thing being measured. A viewport that's slightly off after a resize event. A page that renders an explicit fallback state — which looks superficially fine — instead of the live state a real user would see, because some part of the test setup bypassed the normal loading path. A check whose own self-test silently failed, so every "result" downstream of it is actually noise wearing a verdict's clothes.
Every one of these produces a clean, confident-looking FAIL if your only two options are pass and fail. And a clean, confident-looking FAIL gets treated like a real regression: someone gets paged, someone opens a ticket, someone spends an hour bisecting commits looking for a bug that was never in the application at all. Worse, once a suite has enough of these false fails, people learn to ignore red — which is how a suite stops meaning anything, quietly, long before anyone notices.
UNMEASURED breaks that cycle at the source. If a check can't tell you it observed the right thing, it says so, instead of guessing. That single distinction — "I saw the wrong thing" versus "I saw nothing worth trusting" versus "I saw exactly what I expected" — is the entire value of the third state.
What it costs you, and what it buys you
The honest cost: UNMEASURED means you sometimes don't get a clean answer. A dashboard with three verdicts is less satisfying to look at than one with two, and "6 passed, 1 failed, 3 unmeasured" requires one more sentence to explain than "6 passed, 4 failed." That's a real tradeoff, not a free lunch.
What it buys back is bigger: a FAIL you can trust. When every ambiguous case has somewhere honest to go, a FAIL stops being "something looked wrong, possibly including our own tooling" and starts being "the check got a clean look, and what it saw was wrong." That's the difference between a suite people investigate and a suite people learn to route around.
Where this shows up in QA Reef
Every run in QA Reef reports PASS, FAIL, or UNMEASURED, and every one of those runs keeps a screenshot and a Playwright trace as evidence — so even an UNMEASURED result comes with something to look at, not just a shrug. Bugs are only auto-filed from FAIL, not from UNMEASURED, on purpose: a bug should represent a real failure of a generated test, not a flaky harness making noise. See the Runs surface for where this lives in the product.
Related: Dark controls: testing that a gate says no · Why we read the screen with OCR instead of trusting selectors · Docs: verdict states