Сбор кастомных Prometheus метрик с помощью Proto Observability
Сбор Prometheus метрик в ProtoOBP
Возможны два варианта – скрейпинг метрик агентом или использование remote_write
в Opentelemetry Collector.
Скрейпинг метрик Агентом
Для указания эндпоинтов чтения метрик агенту используйте файл <agent_folder>/conf.d/prometheus.d/conf.yaml
Пример файла:
## All options defined here are available to all instances.
#
init_config:
## @param service - string - optional
## Attach the tag `service:<SERVICE>` to every metric, event, and service check emitted by this integration.
##
## Additionally, this sets the default `service` for every log source.
#
# service: <SERVICE>
## Every instance is scheduled independently of the others.
#
instances:
## @param prometheus_url - string - required
## The URL where your application metrics are exposed by Prometheus.
#
- prometheus_url: http://localhost:10055/metrics
## @param namespace - string - required
## The namespace to be prepended to all metrics.
#
namespace: <SERVICE_NAME>
## @param metrics - (list of mapping or string) - required
## List of `<METRIC_TO_FETCH>: <NEW_METRIC_NAME>` for metrics to be fetched from the Prometheus endpoint.
## `<NEW_METRIC_NAME>` is optional. If set, it transforms the name in Datadog.
## This list should contain at least one metric.
#
metrics:
- <METRIC_TO_FETCH>: <NEW_METRIC_NAME>
- <METRIC_TO_FETCH>
## @param prometheus_metrics_prefix - string - optional
## Prefix for exposed Prometheus metrics.
#
# prometheus_metrics_prefix: <PREFIX>_
## @param health_service_check - boolean - optional - default: true
## Send a service check reporting about the health of the Prometheus endpoint.
## The service check is named `<NAMESPACE>.prometheus.health`.
#
# health_service_check: true
## @param label_to_hostname - string - optional
## Override the hostname with the value of one label.
#
# label_to_hostname: <LABEL>
## @param label_joins - mapping - optional
## Allows the targeting of a metric to retrieve its label with a 1:1 mapping.
#
# label_joins:
# target_metric:
# label_to_match: <MATCHED_LABEL>
# labels_to_get:
# - <EXTRA_LABEL_1>
# - <EXTRA_LABEL_2>
## @param labels_mapper - mapping - optional
## The label mapper allows you to rename labels.
## Format is `<LABEL_TO_RENAME>: <NEW_LABEL_NAME>`.
#
# labels_mapper:
# flavor: origin
## @param type_overrides - mapping - optional
## Override a type in the Prometheus payload or type an untyped metric (ignored by default).
## Supported <METRIC_TYPE> are `gauge`, `counter`, `histogram`, and `summary`.
#
# type_overrides:
# <METRIC_NAME>: <METRIC_TYPE>
## @param send_histograms_buckets - boolean - optional - default: true
## Set send_histograms_buckets to true to send the histograms bucket.
#
# send_histograms_buckets: true
## @param send_monotonic_counter - boolean - optional - default: true
## Set send_monotonic_counter to true to send counters as monotonic counter.
#
# send_monotonic_counter: true
## @param exclude_labels - list of strings - optional
## A list of labels to be excluded.
#
# exclude_labels:
# - timestamp
## @param ssl_cert - string - optional
## If your Prometheus endpoint is secured, here are the settings to configure it:
## Can either be only the path to the certificate where you specify the private key
## or it can be the path to a file containing both the certificate and the private key.
#
# ssl_cert: <CERT_PATH>
## @param ssl_private_key - string - optional
## Needed if the certificate does not include the private key.
## WARNING: The private key to your local certificate must be unencrypted.
#
# ssl_private_key: <KEY_PATH>
## @param ssl_ca_cert - string - optional
## The path to the trusted CA used for generating custom certificates. Set this to false to disable SSL
## certificate verification.
#
# ssl_ca_cert: <CA_CERT_PATH>
## @param prometheus_timeout - integer - optional - default: 10
## Set a timeout in seconds for the Prometheus query.
#
# prometheus_timeout: 10
## @param max_returned_metrics - integer - optional - default: 2000
## The check limits itself to 2000 metrics by default, increase this limit if needed.
#
# max_returned_metrics: 2000
## @param tags - list of strings - optional
## A list of tags to attach to every metric and service check emitted by this instance.
##
#
# tags:
# - <KEY_1>:<VALUE_1>
# - <KEY_2>:<VALUE_2>
## @param service - string - optional
## Attach the tag `service:<SERVICE>` to every metric, event, and service check emitted by this integration.
##
## Overrides any `service` defined in the `init_config` section.
#
# service: <SERVICE>
## @param min_collection_interval - number - optional - default: 15
## This changes the collection interval of the check. For more information, see:
#
# min_collection_interval: 15
## @param empty_default_hostname - boolean - optional - default: false
## This forces the check to send metrics with no hostname.
##
## This is useful for cluster-level checks.
#
# empty_default_hostname: false
## @param metric_patterns - mapping - optional
## A mapping of metrics to include or exclude, with each entry being a regular expression.
##
## Metrics defined in `exclude` will take precedence in case of overlap.
#
# metric_patterns:
# include:
# - <INCLUDE_REGEX>
# exclude:
# - <EXCLUDE_REGEX>
Отправка метрик через OpenTelemetry Collector
ProtoOBP использует возможности remote_write
для получения метрик в формате Prometheus.
Backend сервер по умолчанию принимает метрики по адресу <protoobp-backend-url>:8428/api/v1/write
, где protoobp-backend-url
– URL адрес Proto Backend сервера
По умолчанию порт 8428 закрыт для инсталляции с одним сервером, для инсталляции ProtoOBP с несколькими серверами порт 8428 по умолчанию открыт.
Отправить Prometheus метрики можно с помощью OpenTelemetry Collector
используя Prometheus Remote Write
.
Пример конфигурации для OpenTelemetry Collector
:
exporters:
prometheusremotewrite:
endpoint: "http://<protoobp-backend-url>:8428/api/v1/write"
tls:
insecure: false
Визуализация метрик на кастомных дашбордах
Все собранные Prometheus метрики можно визуализировать на Кастомных дашбордах.