My shared Postgres cluster went 133 days without a single metric being scraped

On 28 August I found that shared-pg, the one Postgres cluster every app in my homelab uses, had never had a metric scraped. It had been 133 days, counting back to the commit that deployed it.

The setup is a four-node K3s cluster, everything applied from a git repo by Argo CD, with kube-prometheus-stack for monitoring and a single CloudNativePG Cluster that every app gets a database on. The Cluster spec has had this in it since April:

spec:
  monitoring:
    enablePodMonitor: true   # kube-prometheus-stack auto-discovers

That comment is mine. It was wrong for the whole time it existed.

How it came up

Earlier the same day an OpenViking ingest retry loop had burned about $2 of a $20 LiteLLM key budget in two bursts of roughly 20 minutes each, and nothing told me. LiteLLM enforces per-key budgets but exports no metrics; the spend state only exists in its Postgres tables, and the first sign a budget has run out is upstream calls failing with BudgetExceededError. So I wrote a set of CNPG custom monitoring queries to turn those tables into cnpg_litellm_* series and alert on them before the money's gone.

When I went to check the new series were landing, there wasn't a single cnpg_* series in Prometheus, new or old.

Why

kube-prometheus-stack ships with a podMonitorSelector that only matches PodMonitors labelled release: kube-prometheus-stack. CNPG generates the shared-pg PodMonitor itself when enablePodMonitor is on, and it stamps its own labels on it. It has no way to add the chart's release label, so the selector never matched and the operator didn't scrape it. There's no error for that; the PodMonitor isn't in the selection, so nothing happens. Nothing about Postgres was wrong, so there was nothing for the cluster to report either.

The fix in the monitoring Application's values:

# before (chart default, never written down anywhere in the repo)
podMonitorSelector:
  matchLabels:
    release: kube-prometheus-stack

# after
podMonitorSelectorNilUsesHelmValues: false
podMonitorSelector: {}

Both keys are needed. NilUsesHelmValues: false on its own leaves the chart-injected selector in place; the explicit empty selector is what overrides it. I know that because I'd been through exactly this already.

On 1 May I'd fixed the same thing for ServiceMonitors. While debugging NetworkPolicies I noticed Immich's server ServiceMonitor had never been scraped since the day it was deployed, for the same reason: the upstream chart doesn't add the release label. Two commits that day, because the first one (serviceMonitorSelectorNilUsesHelmValues: false alone) didn't do it. I opened up ServiceMonitors and didn't touch PodMonitors. shared-pg is the only PodMonitor in the cluster, so it was the only thing that gap could have hit.

The selector wasn't the only thing in the way

Fixing the selector got me a target, and then three more things stopped the metrics arriving, each one hidden behind the last:

With the selector fixed, Prometheus discovered the target and then every scrape came back as connection refused. Every Argo-managed namespace on this cluster is default-deny, and kube-router was rejecting Prometheus on port 9187. A NetworkPolicy in the postgres namespace allowing ingress from the Prometheus pod in observability fixed that. So if you only change the selector, you've gone from "no target" to "target down", which is at least visible.

Then the custom queries themselves. The first time the instance manager loaded the ConfigMap it raced the operator's Role update, got an RBAC forbidden, and dropped the queries while reporting healthy. A content touch on the ConfigMap made it re-read them.

Then permission denied. CNPG 1.29.1 and 1.28.3 moved the metrics exporter off the postgres superuser onto a cnpg_metrics_exporter role that only inherits pg_monitor (CVE-2026-44477). My queries read tables owned by the litellm role, so they need an explicit GRANT SELECT, plus ALTER DEFAULT PRIVILEGES so tables LiteLLM creates later stay readable. The Database CR has no field for table privileges, so the grants are a Job that runs as an Argo CD PostSync hook. Until that ran, the cluster looked healthy, there were no custom series, and the instance manager log had permission denied for table ... on every scrape.

There's one more line in the spec that fails the same way if you get it wrong. customQueriesConfigMap has to list cnpg-default-monitoring explicitly:

    customQueriesConfigMap:
      - name: cnpg-default-monitoring
        key: queries
      - name: litellm-monitoring-queries
        key: queries

The operator's defaulting webhook only injects the stock query set while that field is unset. Add your own entry without it and every standard cnpg_* metric silently disappears.

The failure mode for all of this is silence, so the alert rules I added for LiteLLM spend include an absent() self-check on cnpg_litellm_key_spend_usd. I picked that one because it comes from a table that always has rows, so it should never legitimately be empty. If the scrape stops or the ConfigMap gets dropped, that rule fires after 15 minutes instead of every other rule quietly evaluating against no data.

I still don't have a good answer for the first 133 days. Nothing was looking for those metrics, so nothing could notice they weren't there.