About Kubelet Volume Metrics

Kubelet exposes the following metrics:

1. kubelet_volume_stats_capacity_bytes
2. kubelet_volume_stats_available_bytes
3. kubelet_volume_stats_used_bytes
4. kubelet_volume_stats_inodes
5. kubelet_volume_stats_inodes_free
6. kubelet_volume_stats_inodes_used

Those metrics measure information related to a PVC’s filesystem inside a Longhorn block device.

They are different than longhorn_volume_* metrics, which measure information specific to a Longhorn block device.

You can set up a monitoring system that scrapes Kubelet metric endpoints to obtains a PVC’s status and set up alerts for abnormal events, such as the PVC being about to run out of storage space.

A popular monitoring setup is prometheus-operator/kube-prometheus-stack, which scrapes kubelet_volume_stats_* metrics and provides a dashboard and alert rules for them.

Longhorn CSI Plugin Support

In v1.1.0, Longhorn CSI plugin supports the NodeGetVolumeStats RPC according to the CSI spec.

This allows the kubelet to query the Longhorn CSI plugin for a PVC’s status.

The kubelet then exposes that information in kubelet_volume_stats_* metrics.