Somebody writes a command to fix up old orders. Locally there are 40 of them. It runs in a second.

In production there are 2 million. The command eats all the memory on the box, gets killed, and leaves the data half fixed with no idea where it stopped.

The code wasn't wrong. It just assumed the table would always be small.

get() is a bet on the row count

This is the line that starts it:

Order::where('status', 'pending')->get()->each(function (Order $order) {
    $this->export($order);
});

get() pulls every matching row into memory as a model before the first one gets exported. Fine for 40. A problem for 2 million.

Anything that touches "every record" (exports, backfills, sync jobs, cleanup scripts) should assume the table is bigger than you think. Because eventually it is.

chunk() has a trap in it

Laravel's first answer is chunk(), which runs the query in batches:

Order::where('status', 'pending')->chunk(500, function ($orders) {
    foreach ($orders as $order) {
        $order->update(['status' => 'processed']);
    }
});

This looks right. It isn't.

chunk() pages with an offset. The first batch flips 500 orders to processed, so they drop out of the pending results. Now the second query asks for "pending, skip 500," and the 500 orders that just moved up to the front get skipped. You end up processing about half the rows and the command reports success.

The Laravel docs call this out. If you're updating the thing you're filtering on, use chunkById():

Order::where('status', 'pending')->chunkById(500, function ($orders) {
    foreach ($orders as $order) {
        $order->update(['status' => 'processed']);
    }
});

Instead of "skip 500," each batch asks for "id greater than the last one I saw." Rows changing status doesn't move that marker.

lazyById() if you want a loop

Callbacks get awkward fast. lazyById() gives you the same batching as a plain loop:

foreach (Order::where('status', 'pending')->lazyById(500) as $order) {
    $this->export($order);
}

Under the hood it still fetches 500 at a time. You just don't have to think about it.

There's also cursor(), which runs one query and hands you one model at a time. It uses less memory than get(), but the docs warn that PDO can still buffer the whole result set, so on a really big table it can still run out. For large jobs I'd reach for lazyById() first.

Sometimes you don't need a loop at all

If every row gets the same change, let the database do it:

Order::where('status', 'pending')->update(['status' => 'processed']);

One query. No models in memory.

The catch: this skips model events and observers entirely. If something important lives in a saved hook (cache clearing, audit logs, search indexing), it won't run. Know which one you need before you pick.

Make it restartable

Batching by id has a nice side effect. If the job dies, you know roughly where it was.

Print the last id you finished, and give the command a way to start after it:

$query = Order::where('status', 'pending')
    ->when($this->option('after'), fn ($q, $id) => $q->where('id', '>', $id));
 
foreach ($query->lazyById(500) as $order) {
    $this->export($order);
    $this->line("done through {$order->id}");
}

Now a crash at row 1.4 million means a restart, not a rerun from zero. Pair that with work that's safe to run twice and you can stop babysitting the terminal.

Pages in the UI have the same problem

Offset pagination on a big table gets slow on deep pages, because the database still has to walk past every skipped row. It can also show duplicates or skip records when rows get added while someone is paging.

For feeds and infinite scroll, cursor pagination avoids both:

$orders = Order::orderBy('id')->cursorPaginate(50);

You give up "jump to page 37." For most lists, nobody was doing that anyway.

TypeScript: don't collect every page into an array

Same mistake, different language. Pulling every page from an API into one giant array before doing anything:

const all: Customer[] = [];
let url: string | null = "/api/customers";
 
while (url) {
  const body = await (await fetch(url)).json();
  all.push(...body.data);
  url = body.next;
}
 
for (const customer of all) await sync(customer);

An async generator lets you process each page as it arrives. This assumes an API that returns data and a next link:

async function* paginate<T>(start: string): AsyncGenerator<T> {
  let url: string | null = start;
 
  while (url) {
    const res = await fetch(url);
    if (!res.ok) {
      throw new Error(`${url} failed: ${res.status}`);
    }
 
    const body = (await res.json()) as { data: T[]; next: string | null };
    yield* body.data;
    url = body.next;
  }
}
 
for await (const customer of paginate<Customer>("/api/customers")) {
  await sync(customer);
}

Memory stays flat no matter how many pages there are, and the first record gets synced before the last page is even requested. (In real code, parse that response instead of casting it. The server didn't promise you anything.)

The habit

Whenever code says "for every record," ask two questions:

1. What happens when there are a million of them?
2. If this dies halfway, can I pick up where it stopped?

If the answer is "it loads them all" and "start over," fix that before the table grows. It's a lot less fun to fix after.