Laravel queue monitoring: how to tell when jobs quietly stop

Queues fail in two opposite directions, and most Laravel queue monitoring only looks one way.

• Backlog: jobs are being dispatched but not processed. Workers crashed, a deploy didn't restart them, one slow job is hogging the only worker. Jobs pile up and wait times grow.
• Drought: jobs aren't being dispatched at all. The queue is empty and every worker is idle, which looks exactly like a healthy queue on a quiet day.

Failed jobs are a third case, but they're the easy one: they land in the failed_jobs table and your error tracker hears about the exception.

Watching for backlog

Laravel ships a command for this. Schedule queue:monitor every minute with the queues you care about and a maximum size:

use Illuminate\Support\Facades\Schedule;

Schedule::command('queue:monitor redis:default,redis:emails --max=100')
    ->everyMinute();

When a queue goes over the limit it fires a QueueBusy event, which you can turn into a notification in a service provider:

use Illuminate\Queue\Events\QueueBusy;
use Illuminate\Support\Facades\Event;
use Illuminate\Support\Facades\Notification;

Event::listen(function (QueueBusy $event) {
    Notification::route('mail', '[email protected]')
        ->notify(new QueueHasLongWaitTime(
            $event->connection,
            $event->queue,
            $event->size,
        ));
});

(QueueHasLongWaitTime is your own notification class.) If you run Horizon, it does this for you: set wait-time thresholds per queue in the waits section of config/horizon.php, and Horizon sends a notification when a job has been waiting too long, including to Slack via Horizon::routeSlackNotificationsTo().

Between those two, backlog is well covered. Mostly.

The queue nobody is watching

Here's the gap in backlog monitoring. Say a job is changed to use a new queue:

class SendInvoice implements ShouldQueue
{
    use Queueable;

    public function __construct(public Invoice $invoice)
    {
        $this->onQueue('billing');
    }
}

If your workers run with --queue=default,emails, nothing processes billing. The jobs pile up, but only on a queue that isn't in your queue:monitor list and isn't configured in Horizon. Both tools only watch the queues you've told them about. Nothing errors. Invoices just stop going out.

Watching for drought

Drought is the harder problem, because an empty queue is also what success looks like. Some ways it happens:

• A refactor removes the dispatch() call from an observer, and the job is never queued again.
• An event listener that queued the work is moved to a directory event discovery doesn't scan, so the event fires and nothing responds.
• The code that dispatches jobs sits behind a condition (a feature flag, a config value) that quietly became false.

No monitor that looks at queue size can catch this, because the queue is behaving perfectly. What changed is throughput: how many jobs of each kind get processed per hour.

The simplest place to start is the one job you'd miss most. Don't watch the queue at all; check for the job's effect:

use App\Models\Invoice;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Schedule;

Schedule::call(function () {
    $sent = Invoice::where('sent_at', '>=', now()->subDay())->count();

    if ($sent === 0) {
        Log::critical('No invoices have been sent in the last 24 hours.');
    }
})->dailyAt('09:00');

If invoices go out every day, that will catch the day they stop. It's worth having.

But what about...

...half as many? A listener that stops firing for one payment method doesn't take invoices to zero. It thins them out, and this check never notices.

...every other job? Welcome emails, CRM syncs, stock updates, webhooks out, report generation. Each one needs its own check, its own idea of normal and its own "how long is too long".

...jobs that don't run every day? A weekly statement run or a job that fires a few times a month can't be judged on a 24-hour window.

...waiting until 9am? A daily check means a problem that starts at 9:05 goes unnoticed for a day.

...the deploy that went out this morning? It's the most likely cause, and this check doesn't know it happened.

Answer all of those for every job class in your app and you've built a monitoring system, which someone then has to maintain.

What Marmot does instead

Marmot counts the other side of the queue: what's actually flowing through it.

• Every job class is watched automatically, along with mail and notifications, with no listeners to write. SendInvoice is watched from the first invoice onwards.
• It knows each job's normal rhythm. "SendInvoice has been silent for 5 hours" is an alert on its own, even while every queue is empty and every worker is healthy.
• It works on long-running workers as well as short-lived ones.
• Deploy context. Alerts tell you when something shipped shortly before, which is usually the answer to "why did this stop?"

It doesn't replace Horizon or queue:monitor. Those watch for jobs that are stuck. Marmot watches for the work that stopped arriving.

It sends names and counts only, never job payloads.