Repository navigation
feat(observability): merge the Grafana dashboards into one schema v2 dashboard - #475
Open
gabriele-wolfox wants to merge 11 commits into
Open
gabriele-wolfox wants to merge 11 commits into
gabriele-wolfox wants to merge 11 commits into
Conversation
gabriele-wolfox
force-pushed
the
dev/461
branch
3 times, most recently
from
October 22, 2026 13:29
d794d10 to
acc8f68
Compare
gabriele-wolfox
enabled auto-merge (squash)
October 22, 2026 13:30
gabriele-wolfox
disabled auto-merge
October 22, 2026 13:31
joao00001
reviewed
Oct 23, 2026
joao00001
left a comment
joao00001
left a comment
There was a problem hiding this comment.
Small UX thing: from Klio Clusters there’s no way to see which server backs a cluster, and the dashboard link doesn’t carry the selection over. A small “served by” panel plus includeVars on the link would help, but fine as a follow-up
Comment on lines
+46
to
+47
| WithVariable(labelVariable("cluster", "Cluster", | ||
| `label_values({__name__=~"klio_(plugin|client)_.+",k8s_namespace_name=~"$namespace"}, cluster_name)`)). |
There was a problem hiding this comment.
While testing this, I noticed that with namespace=team-c and cluster=All, the server-side panels show all clusters, not just cluster-c. Picking cluster-c explicitly works fine. I think dropping AllValue(".+") for this variable would fix it, unless it is an expected behavior.
Contributor
Author
There was a problem hiding this comment.
Could you open a PR against this one, so that I can look at the suggestions and eventually apply them?
Thanks a lot!
Comment on lines
+103
to
+104
| WithVariable(labelVariable("cluster", "Served cluster", | ||
| `label_values(klio_server_wal_written_total{service_name=~"$server"}, cluster_name)`)) |
There was a problem hiding this comment.
Since Served cluster comes from klio_server_wal_written_total, a server that doesn’t receive WAL (e.g. read-only mode) won’t list any clusters. Maybe source it from {__name__=~"klio_server_.+"} instead? Same idea as the restore-only fix on the Clusters side
Comment on lines
+145
to
+153
| On the Klio Clusters dashboard: | ||
|
|
||
| - `namespace`: the namespace where the PostgreSQL cluster runs, matched | ||
| against `k8s.namespace.name`. | ||
| - `cluster`: the PostgreSQL cluster, matched against `cluster_name`. It | ||
| lists every cluster that emits Klio plugin metrics, including clusters | ||
| that only restore from a Klio server. | ||
| - `pod` (*Instance*): the PostgreSQL pod, matched against `k8s.pod.name`. | ||
| It scopes the panels built from the plugin metrics. |
There was a problem hiding this comment.
These variables are now described here, but the validation steps in operator/config/samples/opentelemetry/README.md still describe the single dashboard ($server offering klio-a-klio-0, etc.). Might be worth updating them too
|
Aside comment: OLM Bundle & Catalog failed in olm:catalog. This PR doesn’t touch anything under operator/, and the job passes on main, so it looks unrelated. Could you re-run it? |
|
Opened #496 against dev/461 with the suggestions |
gabriele-wolfox
force-pushed
the
dev/461
branch
from
October 26, 2026 12:32
ec850db to
6a60cb8
Compare
Contributor
|
Metrics coming from the server are in the client dashboard. If the distinction between what should be in the client and in the server is not clear, it is worth trying a single dashboard with different filters per group. It has the benefit of keeping a single self-contained dashboard with all the data. It will require Grafana 13.1. |
gabriele-wolfox
changed the title
The single dashboard applied the same namespace, server and cluster filters to every section, but each matched only part of the panels, and clusters that only restore were missing from the cluster list. Replace it with a Klio Clusters and a Klio Servers dashboard, each with filters that apply to all of its panels, and refresh the documentation screenshots. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
On the Klio Clusters dashboard, selecting a namespace and leaving the cluster variable on "All" still showed every cluster on the panels built from the Klio server metrics, because "All" matched any value (`.+`) instead of the clusters of the selected namespace. The server variable of the Klio Servers dashboard had the same problem with the servers. Let "All" expand to the values listed by these two variables, which are already filtered by their parents. The other variables keep matching any value. Source the clusters listed by the "Served cluster" variable from all the Klio server metrics, so a server that does not receive WAL, such as a read-only one, still lists the clusters it serves. Signed-off-by: Joao Detomini <joao.detomini@enterprisedb.com>
…sample The validation steps still described the single dashboard. Split them between the Klio Clusters and the Klio Servers dashboards, and add the checks for the "All" value of the cluster and server variables. Also fix what the sample README said about the server variable: it matches the OpenTelemetry service.name, which the multi sample sets differently for each server, so the two independent klio-a servers are not conflated. Signed-off-by: Joao Detomini <joao.detomini@enterprisedb.com>
The Klio Clusters dashboard did not tell which Klio server retains the backups of a cluster. Group the retained backups panel by server too, and show the server in its legend. The same panel on the Klio Servers dashboard gets the same legend. Signed-off-by: Joao Detomini <joao.detomini@enterprisedb.com>
… time range Now that "All" of the cluster and server variables expands to the listed values, a cluster or server that appears after the dashboard loads stays out of every panel until the page is reloaded. Refresh both variables when the time range changes, which also happens at every auto-refresh. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Two servers reading the same repository, such as a read-only one, both report its snapshots, so the Retained snapshots panel counted them twice. Group it by server too, as the Retained backups panel already does, and widen the Kopia snapshot tiles so the longer legend fits. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Capture the Klio Clusters and Klio Servers dashboards again after the changes to the variables and the per-server snapshot panel. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
…dashboard Replace the Klio Clusters and Klio Servers dashboards with a single Klio dashboard in the schema v2 format, which needs Grafana 13.1 or later. A Clusters and a Servers section each declare their own variables, and every panel query applies all the variables of its section, through a join where the series lack the label a variable selects on. Klio servers are identified by namespace and Server name instead of the OpenTelemetry service name. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Show the retained backups and the backup maintenance in the Servers section, where their series carry the server and cluster labels the filters select on, and keep the time since the last WAL and the WAL bytes written only there, so no panel appears twice. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Capture the Clusters and Servers sections of the single Klio dashboard, rendered with gcx over the last 30 minutes of pgbench load, backups and WAL archiving with tier 2 enabled. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
gabriele-wolfox
force-pushed
the
dev/461
branch
from
October 27, 2026 13:57
d1dda8c to
3ad372d
Compare
Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the Klio Grafana dashboard with a single dashboard in the schema v2 format, which needs Grafana 13.1 or later.
A Clusters and a Servers section each have their own filters, and every filter applies to all the panels of its section. Klio servers are identified by namespace and Server name.
Closes #461
Assisted-by: Claude