Compare

QA Reef vs your own Playwright setup

QA Reef generates Playwright, so this is not an argument against it. It is an argument about the parts around it.

Facts about your own Playwright setup were read from playwright.dev on 30 August 2026 and are described as their published material states them — check the current version before you buy. QA Reef is not affiliated with your own Playwright setup; the name is used here only to identify the product being compared.

The short version

your own Playwright setup in one line

Playwright is Apache-2.0, free, excellent, and has a codegen recorder and a trace viewer in the box. For a lot of teams this is genuinely the correct answer.

QA Reef in one line

An open-core QA agent you deploy like a Vercel project: record or describe a flow, get a deterministic Playwright spec, run it on your hardware or ours, and get PASS, FAIL, or UNMEASURED back.

QA Reef generates Playwright. We are not arguing against it — we are arguing about the parts around it: recording durable locators, deciding what a flaky run means, storing evidence, gating a deploy, and not having a person babysit the suite.

Side by side

	QA Reef	your own Playwright setup
Open source	Open core. The `qareef` CLI is Apache-2.0. The automation core under `packages/core` is MIT; extraction to a public repo is in progress. The generated tests are plain Playwright you already own.	Completely. Playwright is Apache-2.0 and so is its tooling. No vendor, no account, no telemetry you did not opt into.
Self-host	Yes. Node 22 + Playwright + a model key on your own machines, or hosted by us. Per-workspace data roots either way.	Entirely yours. There is nothing else to host.
Pricing model	Not published — pre-launch. Self-hosted you pay only for your own model calls, each logged with its model id and cost. A local-model backend makes that $0.	Free as software. The real cost is engineer time: writing the specs, chasing flakes, keeping the runners alive, and building whatever reporting you end up wanting.
Who writes the tests	You do — by recording in a hosted browser, or by giving the agent a goal. Deterministic codegen writes the Playwright spec; the model only proposes a title and assertions, stamped `decided_by`.	You do, by hand or with `playwright codegen`. Codegen output is a starting point, not a durable suite — the locators still need a human pass.
How flakiness is handled	A heal ladder: wait-and-retry → recorded alternates → DOM heuristic → model proposal → OCR → coordinates. Every heal must clear a same-control gate. Model heals are quarantined for human review, never auto-promoted. A coordinate click, or a run that healed more than 20% of its steps, returns UNMEASURED rather than a pass.	Whatever you build. Usually retries, then a quarantine tag, then someone senior losing an afternoon. There is no built-in notion of a run that could not be measured.
Legacy / canvas UIs	Yes. The operator reads the screen with Apple Vision OCR or a vision model, so canvas and legacy UIs with no stable locators are still driveable.	Hard. If the DOM has nothing to hold on to — canvas, a map, a rendered PDF — you are writing coordinate clicks by hand and hoping the layout holds.
CI integration	`npx qareef` CLI with real exit codes, a GitHub Action, and a Vercel Deployment Check that blocks a promote.	Anything. It is your pipeline.
Data ownership	Self-hosted: flows, screenshots, traces and the model-call ledger never leave your disk. Hosted: per-workspace data root, API tokens hashed at rest.	100% yours. Nothing leaves your infrastructure.

Choose DIY Playwright if…

Your suite is small enough that maintenance is not yet a job.
Your team is strong on Playwright and enjoys owning it. Do not fix that.
You have hard rules against new vendors or outbound model calls.
You want zero abstraction between you and the test. That is a real value.

Choose QA Reef if…

Maintenance has become somebody’s actual job.
You need recording that produces durable locators, with alternates and a real match count, rather than codegen output someone must clean up.
You want a heal ladder with a same-control gate, and a third verdict for runs that could not be measured.
Your UI needs OCR to be driven, or you want a recorded flow exposed as an API.

Where QA Reef falls short

For a small suite, DIY is cheaper, simpler and has no vendor risk. We only start to pay off when the suite is big enough that somebody is spending real time on it — and since QA Reef emits ordinary Playwright, walking away later costs you nothing.

Sitewide caveats, on every page: QA Reef is pre-launch. No customers, no case studies, no published price, no SOC 2 report. Mobile (native app) testing is not supported. There is no managed human-QA service.

Questions

Why not just use playwright codegen?

You should, to start. What codegen does not give you is a durable locator with ranked alternates and a match count, a heal ladder when one breaks, an evidence trail per run, a deploy gate, or a verdict that can say UNMEASURED. That is the gap we build in.

Am I locked in if I use QA Reef?

The output is a plain Playwright spec under your own playwright.config. If you stop using QA Reef, the tests keep running. That is deliberate.

Is DIY actually free?

The software is. The maintenance is not, and it is the part that grows. Price the engineer hours before concluding it is cheaper.

Do I need a model API key?

Not necessarily. There is a local-model backend, so a self-hosted install can run against a model on your own machine at $0 and with no outbound calls.

Sources

Read 30 August 2026. Every statement about your own Playwright setup above comes from one of these; nothing is inferred, estimated, or taken from a third-party comparison.

Comparing more than one?

Every rival we have written up, in one table — QA Wolf, the codeless platforms, the monitoring tools, the agencies, and rolling your own Playwright.

See the full comparison index →