Silent failures: when every check is green and the business is down

Every check is green. The uptime monitor says 100%. The error tracker hasn't had a new issue in days. The queue is empty and the servers are bored.

And nobody has signed up since Tuesday.

This is a silent failure: the app is doing exactly what it was told, returning successful responses, throwing nothing, and the business is quietly not happening. It's the most expensive kind of bug, because every tool you've set up to catch problems is looking for the wrong thing.

Why your monitoring can't see it

Error tracking, APM, uptime checks and log alerts all ask the same question: did something bad happen? An exception, a slow query, a 500, a timeout.

Silent failures aren't something bad happening. They're something good not happening. There's no event to catch, no stack trace to report, no log line to match. The evidence is an absence, and tools built to notice events can't notice absences.

A field guide to failures that never throw

Here are some of the ways it happens in Laravel apps. Every one of these gets past a typical monitoring setup.

The mailer that writes to a log. An environment change leaves MAIL_MAILER=log in production. Every Mail::send() succeeds and writes the email to storage/logs. Password resets, receipts and invitations stop arriving. Nothing errors, because nothing failed.

The listener nobody calls. A listener is moved to a new directory during a tidy-up, outside the paths event discovery scans. The event still fires. Nothing responds. The welcome email, the CRM sync or the stock update it used to trigger simply stops.

The queue nobody listens to. A job is moved onto a new billing queue. The workers run --queue=default,emails. Jobs pile up where no worker, and no queue monitor, is looking. (More in the queue monitoring article.)

Validation that's too strict. A new rule rejects a phone number format a large share of your customers use. They see a polite "The phone field format is invalid" and leave. Validation failures are expected behaviour, so error trackers ignore them by default. Conversion drops and nothing records why.

The front end that quietly broke. A cookie banner update covers the checkout button on Safari. A third-party script fails to load and takes the signup form's submit handler with it. The server sees fewer requests, not failing ones. There's nothing server-side to catch.

The import that succeeds with nothing. The nightly supplier feed returns an empty list. The command runs on time, imports zero rows and exits successfully. (The scheduled task monitoring article covers this one in depth.)

The webhook that stops arriving. Your payment provider couldn't reach your endpoint during an outage, retried for a while, then gave up or disabled it. Subscriptions stop updating. No request arrives, so nothing fails.

The common thread

In every case, something that normally happens stopped happening, or started happening at the wrong rate:

• orders created
• users registered
• emails sent
• jobs processed
• webhooks received
• rows imported

You can't write a test for every way these can break, because most of the causes are outside your code: configuration, third parties, browsers, other people's outages. But you don't need to predict the cause. Watch the outcome. If you know what normal looks like for the handful of moments your business runs on, you can notice when one of them stops, whatever the reason.

A checklist for catching silent failures

1. Name the moments. Write down the five things that, if they stopped, would cost you money or customers. Orders, signups, bookings, renewals, whatever yours are.
2. Know their normal. Roughly how often does each one happen, and how does that change through the day and the week? Monday 10am and Sunday 3am are different. (The anomaly detection article covers what's involved.)
3. Notice when one stops. Silence is the loudest signal you're not listening for.
4. Notice when one slows down. Many failures don't stop a moment outright. They thin it out, and that's harder to spot.
5. Record your deploys. Most silent failures start with a change. Knowing what shipped 20 minutes before the drop turns hours of investigation into one look at a diff.
6. Watch the watcher. If your checks run on your own scheduler, a dead scheduler silences them too. Something outside the app needs to notice.

That's a working definition of "business as usual": not the absence of errors, but the presence of the things that should be happening, at the rate they usually do.

What Marmot does instead

Marmot is built to watch for exactly this, in Laravel apps, with almost no setup.

• It watches the moments your app already produces (new orders, signups, scheduled tasks, queued jobs, emails) without you listing them first.
• It knows what normal looks like for each of them, and tells you when one goes quiet or runs at the wrong rate.
• It's useful from day one, not after weeks of learning.
• It watches your scheduler too, so a dead cron is caught from outside.
• It records your deploys, so an alert says what shipped shortly before it.

Only names and counts leave your app, never your data. Alerts arrive in Slack in plain English: "Orders created has been silent for 3h 12m, expected roughly every 14 min."