The end-to-end suite is slow because it is serial, usually by accident
Before buying more runners, read your own config. In 15 public Playwright suites we counted on 3 September 2026, three run effectively serial in continuous integration, one serializes only its web project, and the large ones shard 4, 8, 10 and 16 ways. The gap between those two groups is not hardware.
Suite counts are ours, taken from public default branches by sparse-cloning the test directories read-only and counting with grep. Statements about how Playwright behaves are quoted from its documentation, read the same day and cited at the bottom. Suites are described without names where the number reads as criticism.
Five checks, cheapest first
- 1. Are your files actually parallel inside themselves? Playwright's documentation is explicit: "By default, tests in a single file are run in order, in the same worker process." A suite made of a few big spec files will not fill your workers until
fullyParallelis set or the describe blocks are configured for parallel mode. - 2. How many workers do you have? The
workersoption "Defaults to half of the number of logical CPU cores", and it can be set as a percentage. On a two-core CI runner that default is one worker, whatever your shard count says. - 3. Is something forcing serial? Look for
workers: 1orfullyParallel: false. Three of the 15 suites we read carry it, one with a comment saying parallelization will come later and a 240-second per-test timeout to survive in the meantime. That is a state-isolation decision wearing a performance costume. - 4. Is your sharding balanced by count or by time? One team moved to duration-weighted sharding across 16 shards, and recorded in its own workflow the reason: count-based splitting stacked the heavy and flaky specs onto a single shard. A large Cypress-era suite chunks 50 ways using a stored timings file, with a dedicated workflow whose only job is keeping the timings current.
- 5. What are retries costing you? 13 of the 15 suites set retries in CI, median 2, maximum 5, and two retry on local runs as well. Retries only cost time on the specs that fail, but a slow flaky spec inside a badly balanced shard is charged two or three times to the critical path.
What the fast suites configure
| Decision | What we counted in the sample |
|---|---|
| Shard count | 4, 8, 10 and 16 for the large Playwright suites. In the Cypress-era sample, 50-way chunking in one suite and 5-way splitting by spec filename prefix in another. |
| Shard balancing | Duration-weighted where the team hit the problem, count-based elsewhere. Nobody balanced by anything cleverer than recorded timings. |
| Job timeout | End-to-end job timeout-minutes from 10 to 120, mostly 15 to 60. One suite runs a 20-minute cap per shard with its own database and mail service containers, and fail-fast: false so one shard does not kill the rest. |
| Artifacts | Uploaded on failure everywhere. Traces set to retain-on-failure or on-first-retry, screenshots only-on-failure. Cheap, and it is the difference between a rerun and a diagnosis. |
| Isolation | The suites that parallelize hardest also seed state through the API or the ORM rather than the interface, and the largest one tags specs with the container capabilities they need, including a database-reset tag. |
One framework difference worth knowing if you are on Cypress: its documentation states that "Running tests in parallel requires the --record flag be passed", so cross-machine parallelization goes through Cypress Cloud, while Playwright shards with a flag in whatever CI you already run. Two suites in our sample hold Cloud project ids and still split work with plain CI matrices. More on that in Playwright vs Cypress, counted.
Parallelism buys wall-clock time, not browser time
Sharding does not change how much browser work exists. It changes how long you wait for it. A suite that uses 160 minutes of browser time still uses about 160 minutes of browser time across 16 shards, plus per-shard startup, and returns in about a tenth of the wall clock.
That matters for how your runner is priced. QA Reef meters browser-minutes at $0.015 past a plan's included allowance, so splitting a run 16 ways costs roughly what running it in one lane costs. Under a per-minute rate, the decision to parallelize is a capacity decision, not a budget negotiation.
For reference on the unit: QA Wolf publishes 15 cents per runner-minute plus 1 cent per AI credit on its self-serve rate card, as published at qawolf.com/pricing on 30 August 2026. Our $0.015 per browser-minute is one tenth of that published per-minute rate. The division is shown in full, with the assumptions stated.
Included allowances, so the arithmetic is easy to do against your own suite: Free $0 with 500 browser-minutes a month plus $20 in signup credit, Solo $20/mo with 1,333, Team $99/mo with 10,000, Scale $499/mo with 60,000 and a priority run queue with burst capacity. AI usage is never metered on any plan. See pricing.
Serial suites are an isolation problem, not a runner problem
Every suite in the sample that runs at one worker could run at eight tomorrow if two tests could not see each other's data. That is the actual work: a fresh tenant, database or account per worker, created over the API or the ORM in a fixture and thrown away afterwards. It is also the work that gets deferred, because a workers: 1 line takes ten seconds and makes the red build go green.
Two ways it shows up in the wild. One suite serializes only its web project while its API tests run parallel, which is a team that diagnosed the boundary correctly. Another carries a 240-second per-test timeout alongside its serial config, which is the cost of the deferral becoming visible.
If you fix nothing else, fix logins and seeding first: 14 of the 15 suites we read never touch the login form in a test, and the largest ones create their fixtures through the ORM. Those two changes usually unlock the workers you already pay for.
Where QA Reef is on this today
Live now
qareef deploysnapshots the flow tree, fans the flows out on the hosted runner under a concurrency cap, rolls the results up, and exits non-zero so your CI job fails.- Runs metered per browser-minute, visible per run, so parallel work does not change the bill.
- Evidence per run: verdict, screenshot, Playwright trace and the flow version that produced it.
- A run that healed too much, or fell back to a coordinate click, returns UNMEASURED rather than a pass, so speed never buys you a fake green.
Next, and shaped by these counts
- Duration-weighted distribution rather than count-based, because that is the problem teams hit at 16 shards.
- A first-class quarantine state: a test that runs and reports but cannot turn the run red.
- Burn-in for newly changed specs, which two teams in the sample built or bought separately.
- Sleep density and skip count reported per suite, since they predict suite health better than the framework does.
Listed as roadmap on purpose. Nothing above is billed for today.
Questions about parallel runs
Why is my suite slow when it already runs in parallel?
Usually because tests inside one file are serial by default, or because workers defaults to half the logical CPU cores and your runner has two. Both are documented Playwright defaults, read 3 September 2026.
How many shards should we use?
The large suites we read use 4, 8, 10 and 16. Pick the number that lands your slowest shard inside your job timeout, then rebalance by duration rather than by file count when the specs stop being similar in length.
Does sharding cost more money?
Not in browser time, which stays roughly constant. It costs CI runner concurrency and per-shard setup. At $0.015 per browser-minute, running 16 shards of the same suite is about the same bill as running one.
Should we just increase retries instead?
Retries are normal, and 13 of 15 suites set them, median 2. But they hide a cost that lands on your critical path, and a retried pass is a different fact from a first-try pass. Track them separately: why end-to-end tests break.
Can we run QA Reef flows in our own CI?
Yes. The CLI has real exit codes and there is a GitHub Action, plus a Vercel Deployment Check that blocks a promote. The generated specs are plain Playwright under your own config, so you can also shard them yourself and never call us at run time. See the platform.
Sources
All read 3 September 2026 unless stated. Suite counts are our own, from public default branches, with the method stated at the top.
- playwright.dev/docs/test-parallel, for parallel files, serial-in-file default and
--shard - playwright.dev/docs/api/class-testconfig, for the
workersdefault - docs.cypress.io, for the recording requirement on parallel runs
- qawolf.com/pricing, as published 30 August 2026, for the 15 cents per runner-minute rate used as a reference point
Sitewide caveats, on every page: QA Reef is pre-launch. No customers, no case studies, no SOC 2 report. Hosted billing has not opened yet. Mobile native app testing is not supported. There is no managed human-QA service.
Send us your playwright.config and a CI log.
We will tell you whether the suite is compute-bound, isolation-bound or just badly balanced, before you buy a single runner.
Talk to the team