Blog
A Flaky Test Is Telling You Something
- testing
- laravel
- typescript
- debugging
Every team has that one test. It fails in CI, someone hits rerun, it passes, and everybody moves on.
Do that enough times and the whole suite stops meaning anything. A red build becomes "probably the flaky one" and real failures slip through wearing its jacket.
A flaky test isn't random. It's depending on something it doesn't control. Your job is to find out what.
The usual suspects
Almost every flaky test lands in one of these buckets:
- Time. The test reads the real clock.
- Order. The test passes alone and fails after some other test runs, or the other way around.
- Shared state. A database row, a cache key, a static property, or a global left behind by somebody else.
- The network. The test is quietly talking to a real service.
- Random data. A factory generates a value that breaks things one run in fifty.
None of those are bad luck. They're inputs you forgot were inputs.
Freeze the clock
If a test calls now() and asserts on the result, it's going to fail at midnight, at month end, or when CI is slow enough that a second ticks over.
Laravel gives you the tools to stop that:
public function test_trial_expires_after_fourteen_days(): void
{
$this->travelTo(Carbon::parse('2026-01-31 10:00:00'));
$user = User::factory()->create();
$user->startTrial();
$this->travel(14)->days();
$this->assertTrue($user->fresh()->trialExpired());
}travelTo() pins the clock and travel() moves it. If you just need time to stop moving, $this->freezeTime() does that.
Pick dates on purpose. January 31 is a great one, because it exposes month math. addMonth() on January 31 overflows into March in Carbon. If you meant "end of next month," you wanted addMonthNoOverflow(), and now you have a test that proves it.
Same idea in TypeScript with Vitest:
import { afterEach, beforeEach, expect, test, vi } from 'vitest';
import { isTrialExpired } from './trial';
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(new Date('2026-01-31T10:00:00Z'));
});
afterEach(() => {
vi.useRealTimers();
});
test('trial expires after fourteen days', () => {
const startedAt = new Date();
vi.advanceTimersByTime(14 * 24 * 60 * 60 * 1000);
expect(isTrialExpired(startedAt)).toBe(true);
});Don't skip the afterEach. Fake timers that leak into the next file are their own flaky test factory.
Shuffle the order on purpose
If a test only passes because another test ran first, you want to find that out on your laptop, not at 4:55 on a Friday.
PHPUnit can randomize order and tell you the seed:
vendor/bin/phpunit --order-by=randomWhen it fails, rerun with the seed it printed:
vendor/bin/phpunit --order-by=random --random-order-seed=1730246400Vitest has the same idea:
npx vitest run --sequence.shuffleOnce you can reproduce the failure with a fixed seed, it stops being flaky. It's just a bug with a repro.
Stop sharing state between tests
The most common order bug is a test that assumes the database is empty, or assumes a row from another test is still there.
In Laravel, use RefreshDatabase and create exactly what the test needs:
use Illuminate\Foundation\Testing\RefreshDatabase;
class InvoiceTest extends TestCase
{
use RefreshDatabase;
public function test_it_lists_unpaid_invoices(): void
{
$unpaid = Invoice::factory()->unpaid()->create();
Invoice::factory()->paid()->create();
$this->getJson('/api/invoices?status=unpaid')
->assertOk()
->assertJsonCount(1, 'data')
->assertJsonPath('data.0.id', $unpaid->id);
}
}Watch for the sneakier stuff too. Static properties, singletons you bound in one test, config you changed with config([...]) in a place that outlives the test, and module-level variables in TypeScript that every test file imports. If it lives longer than one test, it can leak.
Don't assert on an order you never asked for
This one bites constantly:
$this->assertEquals(
['alice@example.test', 'bob@example.test'],
User::pluck('email')->all()
);There's no ORDER BY in that query. The database is allowed to return rows in whatever order it feels like, and it usually matches insert order right up until it doesn't.
Either ask for an order:
User::orderBy('email')->pluck('email')->all();Or say you don't care about one:
$this->assertEqualsCanonicalizing(
['alice@example.test', 'bob@example.test'],
User::pluck('email')->all()
);In Vitest, expect(emails).toEqual(expect.arrayContaining([...])) does the job when order truly doesn't matter.
Cut the network cord
A test that hits a real API fails when that API is slow, rate limited, or down for maintenance. None of that is your code.
Laravel will make stray requests blow up instead of silently going out:
protected function setUp(): void
{
parent::setUp();
Http::preventStrayRequests();
}Now any request you didn't Http::fake() throws. That's annoying for about a day, and then it's the best thing in your suite.
In TypeScript, inject the client instead of importing a global one, so the test can hand in a fake:
type Fetcher = (url: string) => Promise<Response>;
export async function getExchangeRate(currency: string, fetcher: Fetcher = fetch) {
const res = await fetcher(`https://rates.example.test/${currency}`);
if (!res.ok) throw new Error(`Rate lookup failed: ${res.status}`);
return (await res.json()).rate as number;
}test('returns the rate', async () => {
const fakeFetch = async () => new Response(JSON.stringify({ rate: 1.08 }));
await expect(getExchangeRate('EUR', fakeFetch)).resolves.toBe(1.08);
});No mocking library, no global patching, no surprise.
Make random data boring when it matters
Factories are great until fake()->word() happens to generate the one value your slug logic chokes on, or two "unique" emails collide because a test seeds a fixed one too.
If a field matters to the assertion, set it explicitly:
$user = User::factory()->create(['email' => 'owner@example.test']);Random data is fine for fields the test doesn't care about. The moment the test cares, stop rolling dice.
Make it fail on demand
Before you call a flaky test fixed, try to break it. Run it a bunch of times in a row:
for i in $(seq 1 50); do vendor/bin/phpunit --filter=test_trial_expires || break; doneIf it survives fifty runs with shuffled order and a frozen clock, you probably found the real cause. If it fails on run 37, congratulations, you have a repro.
Retries are a painkiller, not a cure
Most test runners let you auto-retry failed tests. Vitest has a retry option and plenty of CI setups bolt one on.
I get why people reach for it. But a retry doesn't fix the test, it hides the evidence. And sometimes the "flaky" part isn't the test at all. It's a real race condition in your code that only shows up under load, and the test was the only thing trying to tell you.
If you have to quarantine a test to unblock a deploy, fine. Put a ticket on it with a name attached. A skipped test with no owner is just a deleted test that hasn't admitted it yet.
The short version
When a test is flaky, ask what it depends on that it doesn't control:
- Freeze the clock and pick nasty dates on purpose.
- Shuffle test order and keep the seed when it fails.
- Start every test from clean state.
- Never assert on an order the query didn't ask for.
- Block real network calls by default.
- Pin any random value the assertion depends on.
- Rerun it fifty times before you call it fixed.
Rerun is not a debugging strategy. The test is telling you something. Listen to it.