Finding the failure behind a green dashboard

Runtime logStart with a known message

Log filterMatch resource and format

Alert policyVerify a positive match

Changing the resource type alone was insufficient; filters also had to match runtime-specific log formats.

Contribution at a glance

My responsibility

Alert inventory, runtime-aware filter repairs and positive-control validation.

Scope

22 alert policies and two log-based metrics affected by a runtime migration.

Collaboration & boundaries

I repaired the monitoring path and checked application logging where a filter change alone was insufficient.

Outcome

Policies and metrics repaired against known log messages; changing only the resource type still returned zero matches.

The problem

After functions moved to a new runtime generation, the application dashboards still appeared healthy. But the log resource type and labels had changed, while alert policies and log-based metrics still filtered the previous format.

An audit identified 22 policies and two metrics in scope. Twenty-one policies were enabled, but the old filters could no longer match their intended events. An absence of alerts had become indistinguishable from an absence of incidents.

My contribution

I traced the migration's effect on resource labels and payload formats, audited related policies and metrics, applied the runtime-specific changes and validated the new filters against log messages known to exist. I also checked the application logging path where changing the filter alone would not restore visibility.

Why the obvious fix was incomplete

Replacing the resource type looked sufficient. It was not: some Go functions emitted structured JSON, while Python functions emitted plain text. A policy using a plain-text field could remain blind even after its resource filter was corrected.

Other policies used a search expression that already matched either payload format. Treating every filter as the same would have introduced unnecessary changes. I verified each shape instead of applying a universal replacement.

Original filter0 matchesResource-type change only0 matchesResource + correct payload field5 matches

Results and measurement

The completion record confirms 22 policies and two metrics corrected and verified. A project-wide sweep found no remaining policy or metric using the obsolete resource filter.

The five-match comparison is a positive-control test for a specific known message. It demonstrates why the resource-only fix was insufficient; it is not a claim that every possible future failure was covered. Some signals were deliberately left disabled where they were redundant, rather than enabling everything indiscriminately.

Extend coverage to the business flow

In a separate follow-up, I found that two high-volume verification scopes were absent from a monitor that watched a much smaller scope. The recorded production snapshot showed approximately 128,000 documents per day across those previously unwatched scopes. I added the missing checks and separated a dormant scope instead of treating one total as proof of complete coverage.

That daily volume describes the observation used to find the gap. It is not a count of failed verifications or prevented incidents.

Distinguish no events from missing state

A later batch-worker investigation separated valid empty event counters from defective state gauges. A healthy circuit breaker must still publish its current state even if it never transitions. The repair also gave each replica an appropriate OpenTelemetry resource identity instead of merging its data into one ambiguous series.

The production validation recorded zero ordering errors in a 3h28m window, compared with errors recurring every minute before the change. The same query returned the earlier errors as a positive control. Event panels could still legitimately have no data when no batch had run; the repair did not manufacture events to make a dashboard look populated.

What I learned

A filter returning zero rows is not a validation result until there is a known event it should match. Migration acceptance needs an observability check that exercises real payloads, not only a successful health endpoint.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

Replacing only the resource type is simpler, but structured Go and plain-text Python logs require different payload matching.

Trade-off

A quiet dashboard cannot distinguish a healthy application from a broken logging or alerting path.

What I would improve next

Next improvement: automate periodic positive-control checks and include alert compatibility in migration acceptance criteria.