Making scheduled jobs observable

Scheduled jobWrap execution and preserve exit code

Persistent signalsPush completion and timestamps

Freshness rulesAlert according to the schedule

24 of 26 active entries were instrumented. Freshness thresholds reflect each schedule; telemetry failures do not replace the job result.

Contribution at a glance

My responsibility

Execution wrapper, read-only exporter, crontab instrumentation and schedule-aware rules.

Scope

24 of 26 active cron entries, 16 Prometheus rules and a 12-panel dashboard.

Collaboration & boundaries

I instrumented job execution without replacing its business logic or real exit code.

Outcome

Persistent completion timestamps made short-lived jobs observable; monitoring failure did not override the job result.

The problem

A legacy cron fleet processed business jobs but exposed no reliable liveness or queue-depth signal. Investigating delivery questions meant searching large log files and writing database queries. A dead job could remain invisible because short-lived processes disappear before a scraper sees them.

My contribution

I built a reusable wrapper, a read-only business-metric exporter and an idempotent crontab instrumenter. The rollout covered 24 of 26 active cron entries and added 16 Prometheus rules and a 12-panel Grafana dashboard.

Make silence measurable

The primary health signal was the timestamp of the last successful execution. A failed or missing run must not advance it. Thresholds were grouped by schedule because a monthly task and a minutely processor cannot share the same lateness budget.

The existing Pushgateway transport was reused. A replacement manifest would have added no useful capability and risked changing an unrelated ingress limit, so I left that infrastructure in place.

Protect the business job from its monitoring

The wrapper propagated the real process exit code. A failed job did not update its success timestamp. When the gateway was unreachable, a successful business job still succeeded. A superseded process receiving SIGTERM was handled separately from an application failure, matching how the existing minutely processors worked.

Results and measurement

The deployment preserved the active crontab entry count, verified the VM-to-gateway path, validated all 16 rules with promtool and checked business processing after the change. Tests covered success, a failing exit code, SIGTERM and an unavailable gateway.

Coverage was intentionally partial: two active entries were outside the wrapper's script model. External provider acknowledgement and actual customer-visible delivery were also kept distinct; this dashboard does not prove an external system displayed every result.

The lesson

Instrumenting legacy jobs is useful when it preserves their behavior. Design the failure path of the telemetry as deliberately as its successful path.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

Scraping an ephemeral process can miss it entirely. Persisting completion signals preserves visibility after exit.

Trade-off

A monthly task and a minutely task need different freshness budgets. Telemetry must not become a new business-job dependency.

What I would improve next

Next improvement: review the two uninstrumented entries and exercise missing-run alerts against each schedule class.