- Prometheus metrics and labels
- Metrics types
- Push and Pull Prometheus model
- Exposing and scraping metrics
- Exporting metrics using side-cars containers
- Summary
Prometheus is one of the most popular open-source systems for monitoring and alerting. Very often it is used to collect data from microservices in Kubernetes. In this series you will learn the main blocks of Prometheus: metrics and labels, push and pull information exchange. You will learn how to use Prometheus without changing the application code. We will generate metrics based on logs (in our case, performance logs).
Prometheus metrics and labels
Metrics are the main concept of Prometheus. Instrumented applications expose them by HTTP endpoint or JMX. Prometheus “scrapes” and stores them in the time-series database. This means that it works very well in a distributed, cloud-native environment. All of the services are unburdened by a load on the monitoring system. Metric is an identifier of data collected by Prometheus. The metric specifies the general feature of a system that is measured - for example http_requests_total - the total number of HTTP requests received. Labels can be added to each data point to specify which service this counter applies to:
Imagine a request counter that looks like this:
http_requests_total{service="spring-app"}
There are many spring-app instances in the cluster so we should add another label - instance:
http_requests_total{service="spring-app", instance="spring-app-1"}
We can add many other labels that accurately describe the request (HTTP method, endpoint, response HTTP status). Metrics with good labels are key to the best use of Prometheus.
Metrics types
Counter
It is the most popular metric type. Counter is an incremental value or zero (after initialization/restart). It is most often used to measure the number of requests, errors.
Gauge
Gauge is the incremental/decremental value that changes over time. It is used for memory or CPU usage statistics.
Histogram
Metrics such as response time are most commonly monitored using histograms. Before starting measurements, we must first define the possible ranges of results. In the case of measuring the response time, it can be: below one second, between 1 second and 2 seconds, between 2 and 10 and above 10. Then we assign the observed value to the matching range.
Summary
The Summary type works very much like a histogram, but it is slightly more extensive, e.g. it stores information about the total number of observations.,
Push and Pull Prometheus model
The monitoring system based on Prometheus can collect metrics in two main ways: push or pull models.
Push
In push model applications are active and they are responsible for sending collected records to Prometheus via Pushgateway. This model works best for gathering information, e.g. with test results, or other cyclical tasks that can be performed irregularly and for a longer period.
Pull
In this model, the application is passive and only prepares its metrics in the form of an endpoint. Prometheus decides when to download them. This approach is recommended by application developers and should be used in most cases.
Exposing and scraping metrics
Most languages have an implementation of Prometheus client with functionalities for:
-
Getting metrics for the language (JVM metrics, GC metrics, Go metrics)
-
Possibility to registry custom metrics
-
HTTP handler to create and expose metrics on the /metrics endpoint
Prometheus scrapes each target regularly (the frequency of meter reading is set by scrape interval property)
Exporting metrics using side-cars containers
Sometimes it is impossible to instrument application at the code level. In this case, the best solution is to use exporters as side-car containers. The most popular are Node exporter and Grok exporter which can read system metrics (CPU, memory…) and expose them for Prometheus.
Summary
You’ve read the basics of Prometheus. You should have a better understanding of what Prometheus is and how it works. You will need this knowledge in the next post. I will show you how to use Grok exporter to instrument Java application based on application logs.