How to adapt Grafana dashboard to changed metrics

Nicolai Antiferov
3 min readJan 3, 2025

--

From time to time happens that either project you’re using is changing metric names or your own services do this and after that you face a dilemma — how to visualize new metrics, but keep possibility to check historical data as well. Examples: metrics rename in karpenter, redis_exporter, nodes_exporter.

One option — you can create Dashboard v2 and keep old metrics in old one and use both. But you need to remember about it, announce change properly in your team/company-wide and worst — it most probably won’t be the last change, so you’ll have to create eventually Dashboard v3, v4 and so on, which is not sustainable.

Another — you can duplicate existing panel in dashboard and change query to a new one. It’s possible to group them in another row or something for better visibility, but it still has same issue — what will happen with next rename — more duplicates/rows?

Third option — setup recording rules and hide mestrics change, but it’s not great either (creates more metrics) and I think makes sense only if you don’t own Grafana dashboard and cannot change it.

Best solution from my perspective would be just use of multiple queries in panels. This way you can not only visualize all renamed metrics in one panel, but also mark them with different legends, so it will be visible on graph if you see old one or new version of metric.

Example: Imagine you’re updating Karpenter to v1 and NodePool usage metric changed name from karpenter_nodepool_usage to karpenter_nodepools_usage.

Before update, you had panel, which showed how many cpu cores used: sum(karpenter_nodepool_usage{resource_type=”cpu”}) by (nodepool, resource_type)

One query

Now you need to clone query A with “Duplicate query” button (second to the right) and modify query accordingly to new naming schema. Also worth to update legend for old query, so it will be visible on graph which version you see.

Two queries, old renamed to legacy

NOTE: If only metric name changed, but not labels, you can also use OR and update only PromQL query, like: karpenter_nodepools_usage or karpenter_nodepool_usage.

That was pretty easy, but what to do if your variable selector depends on metric, which was renamed?

Usually Grafana uses label_values query type and variable definition looks like this: nodepool: label_values(karpenter_nodepool_usage,nodepool)

In order to adapt this variable to support both metric names, you need to slightly modify this query to: nodepool: label_values({__name__=~”karpenter_nodepool_usage|karpenter_nodepools_usage”},nodepool)

This way either of this metrics (old or new) will be used as source for nodepool variable.

NOTE: Please keep in mind that this will require that label nodepool exists in both metrics, so this won’t help in all cases.

--

--

Nicolai Antiferov
Nicolai Antiferov

No responses yet