Flaky Playwright tests: the cost to revenue and releases

You know the scene. CI is green in the morning, red at noon, green again at 2 p.m., and no commit touched the test code. The team reruns the job. It passes. You merge anyway because “it is flaky”. And on a Friday night the release goes out with a confidence that looks more and more like a bet.

This is not a tooling problem. It is a trust problem — and a revenue problem.

Flaky versus a real bug: the difference that costs money

A flaky test fails or passes with no change in code and no controlled change in the environment. A real bug reproduces reliably when the regression is present.

In practice, teams mix the two because CI hides the boundary:

• retries: 2 on GitHub Actions or GitLab CI
• a manual pipeline rerun “just to see”
• a merge despite a “known” intermittent failure

The result: you lose the ability to say no to a release. The pipeline is no longer a guardrail. It is a suggestion.

The flake rate — the share of runs where at least one test fails and then passes on retry — is the most honest signal. Once it is above 5–10% on an E2E suite, you no longer have a test suite. You have a lottery.

Business impact: what flakiness actually costs

Flakiness does not show up on a P&L. It hides in lost time, delayed releases, and incidents you could have avoided.

Releases delayed, or shipped “on a feeling”

Every release blocked by a red CI that nobody can explain pushes value back. Two releases a week, each delayed by half a day, with a team of eight people on standby, is easily several thousand euros of opportunity cost a month — before you count the stress on the product side.

Worse: when trust collapses, some teams bypass CI (“we deploy and we watch”). Revenue can ship faster, but the risk of rollback, a weekend hotfix, and customer churn rises with it.

Hotfixes, rollbacks, production incidents

A flaky test that hides a real regression is a production bug that should have been caught earlier. The cost of an incident — support, comms, rollback, a post-mortem — is far above the cost of a targeted audit.

Invisible debt on the team

Developers and QA spend time:

• rerunning pipelines
• investigating false positives
• justifying merges with no safety net

That time is not spent on the features that generate revenue. It is a silent tax on velocity.

Impact on releases: when the pipeline no longer protects you

A healthy release rests on a clear signal: green means we can ship. With flaky tests:

1. The pipeline blocks — product waits, and release windows get squeezed.
2. Retries hide real failures — you no longer know if it is the test or the app.
3. QA debt becomes invisible — nobody prioritises it because “it usually works”.

The anti-patterns I see most often in an audit:

• fragile CSS locators (.cart-item, div > span:nth-child(3))
• no data isolation between parallel tests
• waitForTimeout instead of assertions on UI state
• specs with no smoke versus regression tags — the whole suite runs for 45 minutes on every PR

You can see what a concrete deliverable looks like in our Audit Flash example: score, dimensions, top flaky tests, prioritised plan.

Prioritise before you rewrite everything

The usual reaction is “throw the suite away and start over”. That is rarely the right move. Often, 20% of the specs cause 80% of the noise.

An approach that works:

1. Measure — flake rate over 30 days of CI runs, not a feeling.
2. List the top 5 flaky tests — the ones that block the most merges.
3. Classify — locator, timing, data, infra, or a real intermittent bug.
4. Fix in waves — quick wins first (locators, explicit timeouts, test ids).
5. Measure again — CI has to become a signal, not noise.

A free scan of your repo already gives a first picture: score, dimensions, anti-pattern signals. The Audit Flash at €490 goes further: an 8–12 page PDF, the top 5 flaky tests, and a remediation plan sized in effort — delivered within 48 business hours, without touching your code.

Why stable tests are non-negotiable before a release

Frequent releases (trunk-based, feature flags, daily deploys) assume a deterministic CI. Without that:

• feature flags ship with no E2E net
• rollbacks become the real quality strategy
• the business loses trust in the engineering team

Investing in stabilisation is not “doing QA a favour”. It protects the delivery rhythm and the revenue.

How Doctests helps with flakiness

I am an independent Playwright SDET. Each engagement uses Doctests, my internal engine: an automated repo scan, correlation with your CI runs when that is relevant, and a human review of every deliverable.

A typical path:

• Audit Flash, €490 — a fast diagnosis, a 0–100 score, the top flaky tests, anti-patterns, and a prioritised plan. Useful when you want an objective picture before you commit a fix budget.
• In-depth flakiness audit — reproduction, root causes, a priced plan. For suites of 100+ specs with a CI that is red several times a week.
• QA Automation Sprint — targeted fixes on the critical flows, a stabilised CI, and a handover on your GitHub.

You keep ownership of all the code. No SaaS lock-in: Doctests is my delivery tool, not a dependency for your team.

If your Playwright CI is red at random, the first step is not to rewrite everything. It is to measure, prioritise, and stabilise. The block at the bottom of the page shows how that can work in practice.