Metrics-driven mastery: How to monitor, alert & scale like a Pro

Click on the backend Deployment, then select Observe tab on the right panel. You can see the CPU usage, Memory usage, and network bandwidth.

To view more monitoring information for a specific Pod in this deployment, click on Resources tab, then click on the Pod to view monitoring details for a particular Pod.

In Pod Details page, select Metrics tab to view monitoring infomation e.g. Memory usage, CPU usage, Filesystem, and Network for this Pod.

To monitor all deployments in this Project, click Observe menu on the left, then select Dashboards tab, default dashboard will show all monitoring information of all Pods in this namespace/Project.

You can view different dashboards by select a dashboard from the Dashboard dropdownlist i.e. Kubernetes/Compute Resources/Pod dashboard to filter and view Pod-specific monitoring information.

The dashboard will show fundamental application monitoring information such as CPU, Memory, Disk, Network etc.

select CPU Usage query, run the query and then select the Stacked checkbox to stack the line chart.

You can use PromQL (Prometheus Query Language) to query metrics information with custom query as well. Delete existing query then type container_cpu in the expression input and wait for auto suggestion.

select container_cpu_usage_seconds_total metric then press Enter on keyboard to query the metric from Prometheus.

Try new query by enter pod:container_cpu_usage:sum to expression input then press Enter to run query.

In the web browser address bar, you can see the URL for the query you’ve just run. You can save or bookmark this URL for later use.

Click Alerts tab to view all alerts (if any). The Alerting UI enables you to manage alerts, silences, and alerting rules. We will create alert later in this lab.

click Events Tab to view All event in this project or filter by resources.

In OpenShift console, click on Search menu on the left, then enter servicemonitor to the Resource dorpdownlist, and then select SM ServiceMonitor from the list.

Click Create ServiceMonitor button.

Delete all YAML code in the YAML editor. Copy this YAML from the code snippet below and paste to the YAML editor in OpenShift console. Then click Create button.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: backend-monitor
  name: backend-monitor
spec:
  endpoints:
  - interval: 30s
    port: http
    path: /q/metrics
    scheme: http
  - interval: 30s
    port: http
    path: /q/metrics/application
    scheme: http
  selector:
    matchLabels:
      app: backend

The ServiceMonitor resource should be created. Click YAML tab to view YAML configuration.

Open the web terminal and run following command to test API calls in backend application 2-3 times. Some custom metric values will be geneated based on these API calls.

oc project userX-observe
curl http://backend.userX-observe.svc.cluster.local:8080

Next try to query the custom metrics. Click on Observe menu on the left, then select Metrics tab.

Enter com to the Expression input and wait for auto suggesstion.

Select com_example_quarkus_BackendResource_countBackend_total from suggestion list or you can type it in expresion input then press Enter on keyboard.

Now, try to query the metric with rate function using PromQL to calcualte per-second average rate of API calls in 1 minute. Copy this query and paste in the expression input then press Enter.

rate(com_example_quarkus_BackendResource_countBackend_total[1m])

Run more curl command in the web terminal and run the metric query again to see the result changes.

curl http://backend.userX-observe.svc.cluster.local:8080

Review this PrometheusRule resource. This resource is used to configure an alert based on the com_example_quarkus_BackendResource_timeBackend_seconds_max metric defined in the expr (expression) field. The alert will be fired when the metric’s value is more than 0.01 (1 second).

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: backend-app-alert
  namespace: userX-observe
  labels:
    openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus
spec:
  groups:
  - name: backend
    rules:
    - alert: HighLatency
      expr: com_example_quarkus_BackendResource_timeBackend_seconds_max>0.01
      labels:
        severity: 'critical'
      annotations:
        message: '{{ $labels.instance }} response time is {{ $value }} sec'

Create the PrometheusRule resource above by click on the + button at the top right corner of OpenShift console then select Import YAML.

Copy the YAML from code snippet above and paste to YAML editor in OpenShift console. Update the userX-observe in the namespace field to matches to your username. Then click Create button.

Run more curl command in the web terminal to trigger API calls.

curl http://backend.userX-observe.svc.cluster.local:8080

Select Observe menu on the left, go to Alerts tab, and wait until the alert shows up. (Refresh web browser if you’ve waited more than 1-2 minutes and don’t see the alert.)

Go to the Topology menu, you should see the alert icon on backend application.

Delete the PrometheusRule resource we’ve created by click on Search menu on the left. Then filter for PrometheusRule resource. Delete the backend-app-alert resource by click on action menu then select Delete PrometheusRule.