Laravel scheduled task monitoring: catch the job that ran and did nothing

A scheduled task can fail in three ways, and most monitoring setups only catch two of them.

1. It never runs. The cron entry went missing when the server was rebuilt. The schedule:run line points at an old release path. A withoutOverlapping() lock was left behind by a crashed run and the task is skipping itself until the lock expires (after 24 hours by default).
2. It runs and throws. An API times out, a column was renamed, memory runs out halfway through.
3. It runs, succeeds and does nothing useful. The supplier feed returned an empty list. A date filter was off by a timezone, so "yesterday's orders" matched nothing. The command exits with status 0 and logs "Imported 0 records".

The third one is the dangerous one, and it's the one this article is mostly about. But first, the two that Laravel already helps with.

Catching a task that never runs

Laravel's scheduler can ping a URL before and after a task:

use Illuminate\Support\Facades\Schedule;

Schedule::command('feeds:import')
    ->hourly()
    ->withoutOverlapping()
    ->pingBefore('https://heartbeat.example.com/feeds-import/start')
    ->thenPing('https://heartbeat.example.com/feeds-import');

Point those at a heartbeat service (Healthchecks.io, Cronitor and Oh Dear all work this way) and tell it the task should check in hourly. If a ping doesn't arrive on time, you get an alert. This is a dead man's switch: it alerts on the absence of a signal rather than the presence of an error, which is the only way to catch something that never happened.

If you'd rather not add ping URLs by hand, spatie/laravel-schedule-monitor records every scheduled task's runs in your database and can sync them to Oh Dear.

One thing to watch: every task needs its own check, and a task added next month needs one too. The failure mode of heartbeat monitoring is a task nobody remembered to wire up.

Catching a task that throws

This one is mostly covered already. Exceptions in scheduled commands are reported through your exception handler, so Sentry, Flare, Honeybadger or Nightwatch will see them. You can add a hook for anything task-specific:

Schedule::command('feeds:import')
    ->hourly()
    ->onFailure(function () {
        Log::critical('feeds:import failed');
    });

The task that succeeded with nothing

Here's the one that gets through. The import ran on time, so the heartbeat pinged. Nothing threw, so the error tracker stayed quiet. The exit code was 0. Every light is green, and your listings haven't updated since Thursday.

The first fix is to make "did nothing" a failure. Count what the task actually did and return a failure code when it's implausible:

public function handle(FeedClient $feed): int
{
    $imported = 0;

    foreach ($feed->items() as $item) {
        Listing::updateOrCreate(
            ['external_id' => $item['id']],
            ['title' => $item['title'], 'price' => $item['price']],
        );

        $imported++;
    }

    $this->info("Imported {$imported} listings");

    if ($imported === 0) {
        $this->error('The feed returned nothing.');

        return self::FAILURE;
    }

    return self::SUCCESS;
}

Then report success and failure separately. Most heartbeat services give you a second URL for failures, and Laravel has a hook for each:

Schedule::command('feeds:import')
    ->hourly()
    ->pingOnSuccess('https://heartbeat.example.com/feeds-import')
    ->pingOnFailure('https://heartbeat.example.com/feeds-import/fail');

Now an empty import is a failed import, and you'll hear about it.

Where that runs out

Zero is a crude line. An import that normally brings in 4,000 rows and brings in 40 still passes. So does a nightly billing run that charges 3 customers instead of 300. To catch those, the task needs to know what a normal run looks like, and now you're keeping history, computing baselines and choosing thresholds per task. (The anomaly detection article covers what that takes.)

It also only covers the tasks you remembered. Output checks live inside each command, heartbeats are configured one task at a time, and neither says anything about the scheduler as a whole.

What Marmot does instead

Marmot watches your scheduler without per-task setup.

• Every scheduled task is watched automatically. A task that stops running raises an alert on its own, with no ping URLs to add and nothing to forget when you add a task next month.
• The scheduler itself is watched too. If cron stops altogether, Marmot notices from outside your server.
• It watches the outcome, not the exit code. Marmot also watches what your app actually produces, so "Listings created" falling from its usual 4,000 a night to 40 is flagged, even though the import reported success.
• Maintenance is understood. php artisan down and up are detected, and alerts are held while you're in maintenance.

It sends names and counts only, never your data, and alerts arrive in Slack in plain English.