Advanced configuration for the Prometheus agent

Prevent billing increases

Avoid sending Prometheus data that isn't relevant to your monitoring needs. Instead, use filters to ignore or include specific metrics, which help you control the amount of data you send to New Relic and avoid additional billing charges.

As explained in the configuration guide, some targets are scraped by default, and all the data that's exposed from those discovered targets is sent. So you may exceed New Relic's platform limits and increase your billing charges.

To reduce the amount of data being set, you can customize the prometheus agent configuration:

• By setting up which targets are being scraped.
• By adjusting the scrape interval.
• By applying metric filters.
• By setting up which prometheus integrations are enabled.

Also, see our troubleshooting tips.

Enable or disable prometheus integrations

The prometheus agent uses the app_values and source_labels fields to scrape metrics from Prometheus integrations, which you can customize. You can check the default values in the chart GitHub repository.

• source_labels defines the label names to be used to filter metrics from the corresponding resources.
• app_values defines the values of those label names which are filtered.

For instance, with the following values:

# (...)
kubernetes:
  source_labels: ["app.kubernetes.io/name", "app.newrelic.io/name"]
  app_values: ["redis"]
# (...)

All jobs with integrations_filter.enabled: true would include only the metrics including the labels corresponding to "app.kubernetes.io/name" and "app.newrelic.io/name", whose value is "redis".

You can extend the default values to include additional labels or application values. You can also take out the values whose services referred in the default application you aren't interested in.

Apply Prometheus native configuration

If you have some kind of requirement which isn't currently supported by the configuration layer the Prometheus agent provides, you can set up additional scrape configuration in Prometheus format. To do so, use the extra_ fields including valid Prometheus configuration which will not be parsed in any way.

Therefore, the metrics collected won't have by default the metadata such as pod or service, added by the agent for the static targets or kubernetes jobs. Use this configuration only as a workaround when the supported fields don't cover your needs. See more details in the helm-chart documentation.