QA when nobody on the team is QA
The usual arrangement: eight to thirty engineers, no QA function, and a release ritual where someone opens the app and clicks the four things that broke last time. It works until the person who remembers the four things is on holiday.
The rules below are copied from suites that large engineering teams actually maintain. We read 15 public repositories with substantial Playwright suites on 3 September 2026, sparse-cloned read-only, and counted with grep: roughly 1,642 spec files and 15,800 test call sites. Where a section is our judgment rather than a count, it says so.
Automate the release ritual, not the test plan
The thing to automate first is the thing you already do by hand every deploy. Not a coverage target, not a test strategy document: the specific clicks that would make you roll back if they failed. For most products that is signup, login, the one action the product is for, the payment or submission step, and whatever broke in the last two incidents.
This paragraph is judgment, not measurement. What follows is measurement.
Five rules, taken from suites that survive at scale
1 · Never log in through the interface
Of the 15 suites, 14 log in over the API or load a saved session file written once by a setup project. Exactly one drives the login form, and one project marks its own UI login helper deprecated in favor of the API version. A login form in every test is the single most expensive habit a young suite can pick up: it is slow, and it makes every test depend on the one screen most likely to change.
2 · Hold on to roles and test ids, not CSS
Role-first is the plurality, 8 of 15. Three lead with data-testid. One is CSS-first and its specs reach into class names that belong to a third-party component library, which is what brittle actually looks like in the wild. XPath is extinct: 12 calls in the entire sample, all in one repository.
3 · Never write a sleep
482 fixed sleeps across roughly 15,800 tests, clustering at 500 and 1000 ms. They hide in fixtures, not just specs: one project's login helper ends with a half-second sleep that every test in the suite then inherits. What the disciplined suites use instead is visible in the counts: about 12,200 auto-retrying visibility assertions. Playwright's own documentation marks waiting for networkidle as DISCOURAGED, saying to "rely on web assertions to assess readiness instead".
4 · Turn retries on and stop apologizing
13 of 15 suites configure retries in CI: median 2, maximum 5, and two retry locally as well. One team runs 0 on principle and writes the reasoning into its config. Retries are not a moral failure, they are a budget. What you owe yourself is the distinction between a first-try pass and a pass on retry, so you can see the trend rather than the average.
5 · Keep the suite small on purpose
Teams that changed frameworks in our sample used it as a coverage audit rather than a port: check whether the old test earns its place, write a replacement if it does, delete it if it does not. One product's suite after that exercise is 43 spec files, standing in for a larger predecessor. Ten tests you trust beat two hundred you rerun.
Bonus · Serial is fine at your size
Three of the 15 suites run effectively serial in CI, one with a comment that parallelization will come later. These are large, well-resourced teams. If a single worker keeps your suite honest while you are small, that is a defensible trade, and it stops being one on the day the run outgrows your patience. When it does: parallel e2e testing.
Decide in advance what a red build means
A team without a QA function has no one whose job is to adjudicate a failure at 6pm on a Thursday, so the answer has to be built into the tooling rather than into a person.
QA Reef's version of that is a third verdict. A run is PASS, FAIL, or UNMEASURED, and the last one is not a euphemism for a pass. A page that never settled, a run that healed more than a fifth of its steps, a step that fell back to clicking a coordinate: all UNMEASURED. Model-proposed fixes to a broken locator are quarantined for a human rather than promoted automatically, so the suite cannot quietly redefine what it is testing while nobody is watching. The reasoning is written up in pass, fail, unmeasured.
The practical effect for a small team is that nobody has to be the judge. A red build means the product is broken. A grey one means the test could not tell you, which is a different task and usually a shorter one.
What this actually costs per month
QA Reef meters one thing: browser-minutes, the compute a real run uses. AI usage is never metered, on any plan.
- Free, $0. 500 browser-minutes a month, plus $20 in signup credit (another 1,333 minutes at the overage rate), no card required.
- Solo, $20/mo. 1,333 browser-minutes included.
- Team, $99/mo. 10,000 browser-minutes, about 25 flows kept green by the agent, deploy checks, 3 seats.
- Scale, $499/mo. 60,000 browser-minutes, priority run queue with burst capacity, unlimited seats, SSO.
- Overage $0.015 per browser-minute on every plan, runs keep going rather than throttling.
- Self-hosted open-source core, $0 with no minute cap. You pay only for your own model calls, each logged with its model id and cost, and a local-model backend makes that zero too.
To size it: a five-flow suite of two minutes each, run on 30 pull requests in a month, is 300 browser-minutes. That fits in the free tier. For reference on the unit, QA Wolf publishes 15 cents per runner-minute plus 1 cent per AI credit on its self-serve rate card, as published at qawolf.com/pricing on 30 August 2026; our $0.015 per browser-minute is one tenth of that published per-minute rate, and the division is shown in full.
Full plan detail on the pricing page.
When you should not do this yourself
Three cases where a tool is the wrong purchase.
- You need someone to decide what to test. Exploratory testing, risk judgment and release sign-off are human work. Automation repeats decisions, it does not make them.
- You need testing as a managed service. QA Reef does not include people. If you want a team to own coverage for you, an agency or a managed QA vendor is the honest answer, and we set out the trade in QA Reef vs outsourced QA agencies.
- You need a compliance artifact by a date. QA Reef is pre-launch: no SOC 2 report, no customer references. If your buyer needs one, we are not the right vendor this quarter.
And if you already have a strong Playwright habit, the honest comparison is against your own setup, including the cases where doing it yourself wins.
Questions from teams with no QA function
How many tests do we need?
Fewer than you think. Teams in our sample treated migrations as coverage audits and deleted what did not earn its place; one product's post-migration suite is 43 spec files. Start with the flows whose failure would make you roll back.
Is it bad if our tests need retries?
No. 13 of 15 public suites configure them, median 2, maximum 5. Just keep a pass on retry visibly separate from a first-try pass, so you are looking at a trend rather than an average.
What does it cost?
Free is $0 with 500 browser-minutes a month plus $20 of signup credit and no card. Solo is $20/mo with 1,333 minutes, Team $99/mo with 10,000, Scale $499/mo with 60,000, overage $0.015 per browser-minute. Self-hosting the open-source core is free with no cap.
Should we hire a QA engineer instead?
Automate the release checks first, because they repeat every deploy and never get more interesting. Hire when the question changes from "did this break" to "what should we be testing", which is a judgment job.
Do we have to write code?
No, but you get code. Record a flow or describe a goal and deterministic codegen writes a plain Playwright spec into your repository. The model only proposes a title and assertions, stamped with the model that decided each one. Walking away later costs nothing, because the specs are ordinary Playwright.
Sources
Suite counts are our own, taken from public default branches on 3 September 2026 with the method stated at the top. Suites are described without names wherever the number reads as criticism.
- playwright.dev/docs/api/class-page, for the discouraged
networkidleload state - playwright.dev/docs/test-retries, for how retries are configured
- qawolf.com/pricing, as published 30 August 2026, for the 15 cents per runner-minute reference rate
- qareef.com/pricing, for our own plans and the browser-minute definition
Sitewide caveats, on every page: QA Reef is pre-launch. No customers, no case studies, no SOC 2 report. Hosted billing has not opened yet. Mobile native app testing is not supported. There is no managed human-QA service.
Bring the four things you click before every release.
We will record one of them with you and hand back a Playwright spec you own, with the evidence from its first run.
Talk to the team