Flaky tests are a team property, not a framework property.
We read the end-to-end suites of 15 products on Playwright and 16 on Cypress and its neighbors, then counted the things that predict flake: hardcoded sleeps, retry policy, worker counts, skipped tests. The spread between teams is 20 to 30 times. The spread between frameworks is not the story. Here is what we found and what to do about it.
What we counted
Two reads, done the same way. Fifteen public products with substantial Playwright suites: roughly 1,642 spec files and about 15,800 test( and it( call sites, read directly including fixtures, page objects, config and CI workflow files. Sixteen public products on Cypress, TestCafe, WebdriverIO or Puppeteer, read the same way, with migration history pulled from commit logs and public issues. Four products appear in both reads because they carry both frameworks.
Everything below is a grep count over a checked-out repository. That means call-site approximations, since parameterized loops distort test totals, and it means these are static proxies rather than observed flake rates. We did not query CI run APIs, so we have no pass-fail history for any suite. Where a number is a proxy, we say so.
Read September 2026. Nothing here comes from a vendor, a survey, or an aggregator.
Same framework, 30x spread in sleep density
A fixed sleep is a bet that the app will be ready in N milliseconds. An intercept-driven wait is a fact. Counting the ratio of one to the other is the cheapest health signal a suite has.
| Cypress suite | Fixed sleeps | Aliased waits | Spec files | Retries in CI |
|---|---|---|---|---|
| Suite A | 820 | 443 | 836 | 0 |
| Suite B | 575 | 83 | 140 | Not counted |
| Suite C | ~316 | 82 | 658 | 1 |
| Suite D | 99 | 10 | 56 | Not counted |
| Suite E | 61 | 1,879 | 424 | 1 |
| Suite F | 6 | 18 | 6 | Not counted |
| Suite G | 1 | 55 | 21 | Not counted |
Fixed sleep counted as cy.wait(<number>), aliased wait as cy.wait('@alias'). Suites anonymized because the point is the spread, not any team. Two rows deserve names for the right reason: Metabase is Suite E, 61 fixed sleeps against 1,879 aliased waits across 424 specs, the discipline outlier of the whole sample. Suite G is Cypress's own reference application, which is what the advertised ideal looks like at 21 specs.
The Playwright side tells the same story with different verbs. 482 waitForTimeout calls across about 15,800 tests, roughly 3 per 100 overall. Per suite, sleeps per 100 tests ran 11, 9, 8, 6, 2, 0.7, 0.7, 0.5, and two suites at effectively zero. That is a twenty-fold spread between the counted extremes, before you reach the two suites at effectively zero, and it lands on the same modal values as the Cypress set: 500 and 1000 milliseconds, with a long tail reaching 5,000 and 6,000.
Two things worth knowing before you go counting.
Sleeps hide in helpers, not just specs. In one suite the shared login fixture ends with a 500 millisecond sleep after its post-login navigation, so every test in the suite inherits it whether or not the spec author ever typed a wait. Grep the fixtures too.
Sleeps also hide behind names. One large suite routes waits through a constants module, so the specs read as named timeouts rather than numbers: 132 uses of a half-second constant, 69 of a one-second constant, 39 of a five-second constant. A grep for a bare number finds none of them. That suite also raises its default command timeout to 30 seconds, which is 7.5 times the framework default, and that is its own signal.
The five things that actually cause it
Ordered by how often we saw evidence of each, and paired with the fix the disciplined suites in the sample already use.
1 · A sleep standing in for a condition
The most common and the most fixable. A fixed wait passes on a fast machine and fails on a loaded CI runner, which is exactly the definition of flake. It is also self-reinforcing: the fix for a failing sleep is usually a longer sleep.
Fix: assert the state you were waiting for. Across the Playwright sample, toBeVisible is the dominant assertion by a wide margin, about 12,200 calls, because it auto-waits. waitForURL shows up about 470 times. For genuinely eventual consistency, expect(...).toPass() and expect.poll exist; one suite alone uses them 317 times.
2 · State bleeding between tests
The expensive one. Two tests share a user, a record or a tenant, and order starts to matter. The measurable tell is what teams do about it: three of the 15 Playwright suites run a single worker in CI, one with a comment saying parallelization will come later, and a fourth serializes just its web project. Serialization is what teams reach for instead of fixing isolation.
Fix: make setup cheap enough to do per test. The suites that parallelize well all seed through the API or the ORM and never through the UI. One resets from database snapshots 1,236 times across its suite; another tags specs with the container capabilities and database reset they need, so the runner can give each test a clean world.
3 · Waiting on the wrong signal
Network idle is the classic. One suite carries 194 waitForLoadState('networkidle') calls, a wait Playwright's own documentation discourages, because a page with polling or analytics never goes idle and a page that goes idle early is not necessarily ready.
Fix: wait on the specific response or the specific element. The discipline outlier in the Cypress set has a 30 to 1 ratio of intercept-driven waits to fixed sleeps, and that ratio is the whole difference. On Playwright, waitForResponse does the same job.
4 · Selectors coupled to copy or to someone else's CSS
Two distinct failure modes, both visible in the counts. In one suite, text queries at 6,172 uses outnumber test-id queries at 3,398, which means a marketing copy change breaks tests. In another, thousands of raw CSS selectors include class names from a third-party component library, so a dependency bump breaks tests.
Fix: a ranked locator policy. Where the app has a test-id convention, prefer it. Otherwise role plus accessible name, then label or placeholder, then text scoped inside a container, and CSS last. XPath is effectively extinct: 12 calls in the entire Playwright sample, all in one repository.
5 · A nondeterministic backend
If the thing under test calls a model, a payment provider or a third-party API, the test inherits that variance. This is the newest cause and the least covered by conventional advice.
Fix: replace it at the boundary. LibreChat's end-to-end setup directory contains fake model and fake MCP servers plus record and replay of model responses, and 63 of its 79 specs live under a mock directory. If you test an AI product, that is the shape of a stable suite.
And one honorable mention: motion
Animations and transitions produce the near-miss click and the assertion that fires one frame early. At least one suite in the sample disables animation globally, both through a reduced-motion context option and by injecting a cookie the app reads.
One suite also ships an anti-flicker assertion worth stealing: a helper that polls to confirm an element does not disappear for N seconds, which catches the element that renders, vanishes, and renders again.
Retries are normal. They are also not the fix
Of the 15 Playwright suites, 14 set CI retries above zero. The full distribution, read out of the configs: one suite at 0, three at 1, six at 2, one at 3, two at 4, two at 5. Median 2, maximum 5. Two teams retry locally as well.
The single team that sets zero says why, in the config, and it is the clearest statement of the opposing view we found. Ghost, MIT licensed:
retries: 0, // Retries open the door to flaky tests.
// If the test needs retries, it's not a good test
// or the app is broken.Both positions are defensible. What is not defensible is treating a retry count as the flake strategy, and the sample shows why.
What the teams with retries still had to build or buy
- One pays for a hosted flake dashboard and rebuilt its CI on duration-weighted sharding, because count-based sharding kept stacking the heavy and flaky specs onto one shard.
- Another routes visual tests to a dedicated visual service rather than trusting in-repo screenshots. In-repo visual regression is nearly absent across the sample: a couple of screenshot assertions per suite at most.
- Mattermost runs a self-hosted automation dashboard service that orchestrates parallel spec distribution and test cycles in CI.
- Metabase chunks CI 50 ways balanced by a stored timings file, keeps two stress-test workflows that re-run changed or flagged specs repeatedly, and uses a pull request label to trigger burn-in on exactly the specs that pull request touched. It is the most disciplined suite in the sample and it still needs all of this.
- PostHog wrote a custom Playwright quarantine reporter after migrating: quarantined tests still run and still report, but their failures cannot turn the run red, and the logic is deliberately conservative about crashes and global errors.
Read that list again as a product statement. Timing-balanced sharding, changed-spec burn-in, and a real quarantine state that runs and reports without blocking are things every team at scale in this sample either built by hand or bought around. Retries bought them a green build, not a working signal.
The graveyard tells you where the flake went
Across the Playwright sample: 380 test.skip and 94 test.fixme, about 3 percent of all tests. Most of the skips are legitimate and well-labeled environment gates. Suites skip on billing being enabled, on an enterprise license flag, on a payment or auth provider not being configured. That is good engineering and it should not be read as rot.
The fixme markers are different. Three suites carry 35, 30 and 28 of them, many as a bare marker at the top of a describe block with a TODO comment under it. In one repository the same comment about a test being too flaky and blocking pull requests is copy-pasted across at least four files, which is a small archaeological record of a problem nobody had time to fix.
No suite in the entire sample keeps an in-repo quarantine list. The fixme pile is the de facto quarantine, and unlike a real quarantine it does not run, does not report, and has no owner or expiry date.
The rules teams write, and then break
Several suites now ship agent instruction files inside the test package, listing anti-patterns for both humans and AI assistants. One such file explicitly names waitForTimeout() as flaky and tells you to use a response wait or a visibility assertion instead. The same suite still contains 13 of them.
That is not hypocrisy, it is the normal gap between a written rule and an enforced one. The suites that closed the gap did it with a static analyzer in CI, not with documentation. One team built a linter for its own test package by hand for exactly this reason.
Worth noting on the other side: forbidOnly in CI is near-universal across the sample, and we found zero committed .only calls in any suite. Rules that CI enforces get followed.
Score your own suite in about a minute
These are the exact counts we ran. None of them is a flake rate. Together they rank your specs by risk before you have any run telemetry at all.
Playwright
# fixed sleeps, in specs AND in fixtures
grep -rn "waitForTimeout(" tests/ | wc -l
# the wait that lies to you
grep -rn "networkidle" tests/ | wc -l
# the graveyard
grep -rnE "test\.(skip|fixme)\(" tests/ | wc -l
# what you assert on
grep -rc "toBeVisible" tests/ | awk -F: '{s+=$2} END {print s}'
# and read these two out of playwright.config
grep -nE "retries|workers|fullyParallel" playwright.config.*Cypress
# fixed sleeps vs intercept-driven waits
grep -rnE "cy\.wait\([0-9]" cypress/ | wc -l
grep -rn "cy.wait('@" cypress/ | wc -l
# sleeps laundered through a constants module
grep -rnE "cy\.wait\(TIMEOUTS?\." cypress/ | wc -l
# the graveyard
grep -rnE "\.(skip|only)\(" cypress/e2e | wc -l
# and read defaultCommandTimeout out of the config
grep -n "defaultCommandTimeout" cypress.config.*| Signal | What we observed in the sample | How to read your own number |
|---|---|---|
| Fixed sleeps per 100 tests | 11 at the top, under 1 for the disciplined suites, about 3 overall | Above 5 and sleeps are your first project, ahead of anything else on this list |
| Fixed sleeps to aliased waits | From 10 to 1 in one direction, to 1 to 30 in the other | The ratio matters more than either count. Push it below 1 to 1 |
| Long sleeps | Modal values 500 and 1000 ms, tail to 5,000 and 6,000 | Anything above 2,000 ms is load-bearing and should be an assertion |
| Skips and fixmes | About 3 percent of tests, most skips legitimately gated | Separate them. Environment gates are fine, bare fixmes are debt |
| CI retries | 14 of 15 above zero, median 2, max 5 | 1 or 2 is unremarkable. 4 or 5 means the retries are load-bearing |
| CI workers | 3 of 15 run one worker | One worker means state isolation is unsolved, not that your suite is small |
| Default timeout | One suite raised it to 7.5x the framework default | A raised global timeout hides slow specs. Raise per assertion instead |
Two things we could not determine, and neither can anyone reading a repository from the outside: real wall-clock suite durations, and actual pass rates. Config files show timeouts and shard counts, not history. Treat every number above as a proxy, including ours.
Three commitments that come straight out of the counts
- The generator never emits a sleep. A recorded step becomes a web-first assertion, or a
toPassblock where the app is genuinely eventually consistent. There is no code path in our generator that writeswaitForTimeout, because 482 of them in someone else's suite is a good argument that the tool should not make it easy. - Locators are ranked, not scraped. Test-id where the app has a convention, then role plus accessible name, then label or placeholder, then text scoped inside a container, then CSS. Locators descend into iframes, pierce shadow DOM, reject generated ids, and carry a live uniqueness count. No XPath, and never a bare pair of coordinates.
- A run that could not be measured returns UNMEASURED, not a pass. This is the part that matters most for flake. A coordinate click, a run that healed too much, a page that never settled: each of those is a third state, reported as itself. Model-proposed heals are quarantined for a human and never auto-promoted. A green build that was green for the wrong reason is worse than a red one, because you stop looking. That argument is written out in full on why UNMEASURED exists.
What we are still building, stated as roadmap rather than shipped: importing an existing Cypress or Playwright suite and reporting its health on the way in, flake scoring across runs, burn-in for new specs, duration-weighted sharding, and a first-class quarantine state that runs and reports without blocking a merge. Pre-launch, and marked as such everywhere on this site.
Questions about flaky tests
What actually causes flaky end-to-end tests?
In the suites we read, four causes dominate. Hardcoded sleeps standing in for a real condition. State bleeding between tests, which teams work around by dropping to a single CI worker rather than fixing isolation. Waiting on the wrong signal, such as network idle, instead of on the assertion you care about. And selectors coupled to copy or to third-party CSS class names. Nondeterministic backends are a fifth cause for AI products, and the one suite in our sample that handled it well replaced the model with a fake server and record-replay.
Is Playwright less flaky than Cypress?
Not by itself. We counted fixed-sleep density in both frameworks and the spread is 20 to 30 times between teams within the same framework, which is far wider than any gap between frameworks. Playwright's web-first assertions make the right thing easier to write, but the disciplined and undisciplined suites in our sample both exist on both frameworks. Flake tracks team practice. See also the Cypress to Playwright migration notes, where every migrated team rebuilt its flake tooling on the far side.
How many retries should I set in CI?
Retries in CI are the norm rather than the exception. Of the 15 Playwright suites we read, 14 set CI retries above zero, the median is 2 and the maximum is 5. Two teams also retry locally. So 1 or 2 is unremarkable. What matters more is that retries are a reporting decision, not a fix: several of these teams pay for or built flake tooling on top of their retries, which tells you the retries were not solving the problem.
How do I measure how flaky my suite is without run history?
Use static proxies you can count in seconds. Fixed sleeps per 100 tests, the ratio of fixed sleeps to intercept-driven waits, the number of skipped and fixme tests, the CI retry count, and the CI worker count. None of these are flake rates, but together they rank your specs by risk, and they are available before you have any telemetry at all. In our sample the highest-density suite ran about 11 fixed sleeps per 100 tests and the lowest disciplined ones ran under 1. The commands are above.
Why do teams run end-to-end tests single-threaded?
Because state isolation is unsolved and serial execution hides it. Three of the 15 Playwright suites we read run one worker in CI, one of them with a comment saying parallelization will come later, and a fourth serializes just its web project. Serialization is what teams do instead of fixing per-test isolation. It works, and it costs you the wall-clock time that parallel execution was supposed to buy.
What is test quarantine and does it help?
Quarantine means a known-flaky test still runs and still reports, but its failure cannot turn the run red. PostHog built exactly this as a custom Playwright reporter after migrating, and made it deliberately conservative about crashes and global errors so a genuinely broken run is not swallowed. It helps, on one condition: quarantine has to be a visible state with an owner and a date, not a place tests go to die. We found no in-repo quarantine lists anywhere else in the sample; the closest thing was accumulated fixme markers.
Is any of this a criticism of the teams whose code you counted?
No, and we have tried hard to write it so it cannot be read that way. Every suite here is a large open-source product solving real problems in public, which is the only reason anyone can count anything. Where a number is unflattering we report it without a name. Where a team did something we admire we say who. Every product name belongs to its owner and appears only to identify a public repository.
Want these counts run against your suite?
Bring a repository or just the numbers from the commands above. We will tell you which specs to fix first and why, whether or not you ever buy anything.
Talk to the team