Excessive CPU or memory consumption

Problem

You have installed the Prometheus OpenMetrics integration for Docker or Kubernetes, and it consumes too much memory or CPU.

Solution

When running the integration in a huge cluster scraping hundreds of targets, CPU and memory consumption will increase, and the number of workers could affect scrape_duration.

For example, a Prometheus OpenMetrics integration consumes 2.5 CPU and 700Mb of RAM because:

• It scrapes 800 targets, exposing 1000 timeseries each.
• Each one has a latency of 150ms with a scrape_duration of 30 seconds.

To reduce resource consumption:

1. Update the integration to the latest available image.

2. Reduce harvest time by lowering emitter_harvest_period. (The default value is 1s, and the interval cannot be smaller than 200ms.) Since metrics are sent more often, memory consumption is reduced.

3. Collect metrics less frequently by increasing scrape_duration to reduce both memory consumption and CPU usage.

4. Reduce the number of workers to reduce both memory consumption and CPU usage. Scraping will slow down and could exceed scrape_duration. To do so:

   • Update the integration to the latest version available of the image.
   • Decrease worker_threads from the default value of 4 to your preferred value.