Prometheus: Basic Commands to Know

Prometheus is a powerful, lightweight, scalable, and easy-to-use Kubernetes monitoring tool, typically used by developers and system administrators. You can use Prometheus to collect data and categorize issues. You use this data to configure a system for prioritizing ops and security tasks. In this article, you will learn about the main advantages of using Prometheus for Kubernetes monitoring, and what are the main queries and functions you can use.

Advantages of Prometheus

Advantages of using Prometheus include:

• Flexible data model—Prometheus stores data as time series. The data is stored as sequences of timestamped values belonging to the same metric. This flexible data model enables you to discover problems in real-time without recreating the issue outside of the system after the event.
• Native alert management—you can configure notifications according to your own rules and requirements. This feature can help you to avoid the use of external APIs or systems.
• Kubernetes integration—Prometheus is the default monitoring tool used by Kubernetes. It supports monitoring of dynamically scheduled services and service discovery.
• Pull based metrics—proactively collects metrics. Prometheus pulls the metrics from exposed HTTP endpoints.

Prometheus: Querying Basics

The Prometheus Query Language (PromQL) enables users to select and aggregate time series data in real time. You can show the result of an PromQL query as a tabular data in Prometheus's expression browser as a graph, or you can transfer the result to external systems via the HTTP API.

Data types of the expression language
The expression language in Prometheus can evaluate the following data types:

• String—a simple value that represents text.
• Scalar—a simple value that represents numeric floating points.
• Instant vector—a set of time series data containing a single sample for each time series. The data shares the same timestamp.
• Range vector—a set of time series data containing a range of data points over time for each time series.

Only some of these data types can be a valid result of user-specified expression. It depends on whether you want to display the output as a graph or not. For instance, you can directly graph only the expression that returns an instant vector.

String characters
You can specify strings as characters in backticks, double quotes or single quotes. The escaping rules of PromQL are similar to the rules of Go. A backslash in single or double quote begins an escape sequence. The sequence may be followed by a, b, f, n, r, t, v or . You can provide specific characters using hexadecimal (\xnn, \unnnn and \Unnnnnnnn) or octal (\nnn).

Prometheus does not discard new lines inside backticks. You can process no escaping inside backticks. For example:

"this is a regular string"
`this string is not unescaped: \n ' " \t`
'This string is unescaped: \n \\ \t'

Instant vector selectors
Instant vector selectors enable you to select a set of time series and a single sample value for each time series at a given timestamp. This enables you to specify the metric name in a simplest form. As a result, you can create an instant vector that contains elements for all time series with this metric name.

For example, you can select all time series that have the total_http_requests metric name:

total_Http_requests

You can further filter these time series by adding a comma separated list of label matchers in curly brackets ({}).

The query below selects only the time series that have the total_http_requests metric name, the group label set to canary, and the job label set to prometheus:

total_http_requests{job="prometheus",group="canary"}

Offset modifier
You can change the time offset for range vectors or individual instants in a query with the offset modifier. For instance, you can return the value of total_http_requests from 5 minutes ago.

Total_http_requests offset 5m

Make sure that the offset modifier follows the selector.

The following query would be correct

sum(total_http_requests_{method="GET"} offset 5m)

The following query would be wrong

sum(total_http_requests{method="GET"}) offset 5m

You can also use offset modifiers in range vectors. This following command returns the 5-minute rate that the metric had a week ago:

rate(total_http_requests_[5m] offset 1w)

Basic Functions of Prometheus

Prometheus offers some basic functions for monitoring.

delta()
delta(vec range-vector) returns the difference between the first and last value of time series elements. The output is an instant vector with the calculated difference and equivalent labels. You can get a non-integer result despite the fact that the sample values are all integers because the delta function covers the full time range in the range vector selector.

The following example calculates the difference in CPU temperature between now and two hours ago:

delta(temp_cpu_celsius{host="zeus"}[2h])

resets()
resets(vec range-vector) calculates the number of counter resets for each input time series within the provided time range. A counter reset is a decrease in the counter value between two successive samples. You should use resets only with counters.

rate()
rate(vec range-vector) returns the average rate of increase of the time series. The following command calculates the per-second rate of HTTP requests as measured over the last 5 minutes in the range vector:

rate(total_http_requests{job="api-server"}[5m])

You should use rate only with counters. Counters are well suited for graphing of slow-moving counters and alerting.

abs()
abs(vec instant-vector) returns the absolute value of all sample values in the input vector

Conclusion

Prometheus can help you gain better visibility into your systems and tools. Prometheus scrapes metrics from agents running on the target systems and stores the collected metrics on its server. Use the commands and functions mentioned above to properly configure your monitoring processes.