Collecting Metrics in Knative¶

Knative leverages OpenTelemetry for exporting metrics. We currently support the following export protocols:

You can also set up the OpenTelemetry Collector to receive metrics from Knative components and distribute them to other metrics providers that support OpenTelemetry.

Note

The following monitoring setup is for illustrative purposes. Support is best-effort and changes are welcome in the Knative Monitoring repository By default metrics are exporting is off.

About the Prometheus Stack¶

Prometheus is an open-source tool for collecting, aggregating timeseries metrics and alerting. It can also be used to scrape the OpenTelemetry Collector that is demonstrated below when Prometheus is used.

Grafana is an open-source platform for data analytics and visualization, enabling users to create customizable dashboards for monitoring and analyzing metrics from various data sources.

Prometheus Stack is a preconfigured collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, combined to provide end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. The stack includes by default some Prometheus packages and Grafana.

Setting up the Prometheus Stack¶

Install the Prometheus Stack by using Helm:

```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install knative prometheus-community/kube-prometheus-stack --create-namespace --namespace observability -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/promstack-values.yaml

Access the Prometheus instance locally¶

By default, the Prometheus instance is only exposed on a private service named prometheus-operated.

To access the console in your web browser:

Enter the command:

kubectl port-forward -n observability svc/prometheus-operated 9090:9090

Access the console in your browser via http://localhost:9090.

Access the Grafana instance locally¶

By default, the Grafana instance is only exposed on a private service named prometheus-grafana.

To access the dashboards in your web browser:

Enter the command:

kubectl port-forward -n observability svc/knative-grafana 3000:80

Access the dashboards in your browser via http://localhost:3000.

Use the default credentials to login:

username: admin
password: prom-operator

Enabling Metric Collection¶

To enable prometheus metrics collection you will want to update config-observability ConfigMap and set the metrics-protocol to prometheus. For request-metrics we recommend setting up pushing metrics to prometheus. This requires enabling the Prometheus OLTP receiver. This is already configured in our monitoring example.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
data:
    # metrics-protocol field specifies the protocol used when exporting metrics
    # It supports either 'none' (the default), 'prometheus', 'http/protobuf' (OTLP HTTP), 'grpc' (OTLP gRPC)
    metrics-protocol: prometheus

    # request-metrics-protocol
    request-metrics-protocol: http/protobuf
    request-metrics-endpoint: http://knative-kube-prometheus-st-prometheus.observability.svc:9090/api/v1/otlp/v1/metrics

    tracing-protocol:      http/protobuf
    tracing-endpoint:      http://jaeger-collector.observability.svc:4318/v1/traces
    tracing-sampling-rate: "1"

Apply the ServiceMonitors/PodMonitors to collect metrics from Knative Serving Control Plane.

kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/serving-monitors.yaml

Import Grafana dashboards¶

Grafana dashboards can be imported from the monitoring repository.
If you are using the Grafana Helm Chart with the dashboard sidecar enabled (the default), you can load the dashboards by applying the following configmaps.
```
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/configmap-serving-dashboard.yaml
```