The service is running.

That sentence is often useless.

A process can be active in a service manager while it is repeatedly dying, restarting, and never becoming useful. A port can accept a connection while every request returns an error. A dashboard can display green because it asked the cheapest possible question: does something exist?

That is the zombie-service problem. It is worse than a clean crash because it produces reassuring noise.

State is not health

A process state answers a narrow operational question: is the supervisor currently managing a process for this unit? It does not answer whether the application completed startup, can reach its dependencies, serves the expected response, or has been stable for any meaningful amount of time.

Those are different questions. Treating them as one is how a failing service gets a green badge.

A restart policy is useful. Transient failures happen, and automatic recovery beats waking someone up for every hiccup. But a restart policy can also turn one failure into an infinite loop:

  1. the process starts;
  2. it exits before readiness;
  3. the supervisor starts it again;
  4. the status page sees an active unit somewhere in that cycle;
  5. nobody notices the application never became healthy.

The service is not recovering. It is rehearsing failure.

The signals that matter

I want three kinds of evidence before I call a service healthy.

1. Restart count

For a systemd unit, inspect the unit rather than stopping at is-active:

systemctl --user show example.service \
  -p ActiveState \
  -p SubState \
  -p NRestarts \
  -p ExecMainStatus

NRestarts is a cheap high-signal counter. It has no universal “bad” threshold: a short-lived worker may restart by design, while a web service should normally stay up. What matters is a baseline and an alert when the counter moves unexpectedly.

A restart counter by itself is not a diagnosis. It is a reason to look.

2. Readiness, not mere liveness

A TCP connection proves that something answered the network stack. It does not prove the application can do its job.

Health checks should test the smallest user-visible operation that captures readiness. Depending on the service, that could mean an HTTP endpoint with an expected status and body, a queue consumer that has registered successfully, or a command that completes one harmless query.

Avoid turning a health endpoint into a fantasy novel. It does not need to validate every dependency on every request. It does need to fail when the thing users need is unavailable.

3. A stability window

A service that becomes ready for a moment and immediately falls over is not healthy. Record how long a successful process has been alive and alert on repeated short lifetimes.

The point is not to make monitoring elaborate. The point is to distinguish a recovery from a loop.

Why green dashboards lie

Dashboards gravitate toward binary signals because binaries are easy to render. Green is emotionally efficient. It is also easy to game accidentally.

  • Unit active: may only mean the supervisor is still trying.
  • Port open: may only mean a proxy or listener exists.
  • Process present: may only mean the failure has not happened yet.
  • Recent log output: may only mean the application is loudly failing.

None of those are worthless. They are inputs. They become misleading when they are presented as the conclusion.

A useful dashboard makes the distinction obvious: current state, restart trend, last successful readiness check, and a link to the relevant logs. The operator should not have to infer a restart storm from a green dot and a growing electricity bill.

A minimal check worth keeping

For a service you expect to run continuously, make this boring routine:

systemctl --user show example.service \
  -p ActiveState -p SubState -p NRestarts -p ExecMainStatus

systemctl --user status example.service --no-pager

Then pair it with an application-specific check. For an HTTP service, check the expected response rather than only whether a socket opens. For a worker, check the work it is meant to perform. For a database-backed application, verify one safe operation through the application path.

When the restart count rises, read the first failure, not the thousandth repetition. The useful log line is usually near the original startup error: a missing dependency, invalid configuration, exhausted permission, unavailable upstream, or a process that was never meant to be supervised as a permanent service.

The actual lesson

A crash is honest. It fails loudly enough that a human usually notices.

A zombie service is more dangerous because it looks like resilience while it burns cycles and trust. The supervisor is doing exactly what it was told; the monitoring is asking the wrong question.

Watch the restart counter. Check readiness. Require stability.

Then green means something.