NVIDIA Triton integration

Our NVIDIA Triton integration monitors the deployment and management of AI models in production environments. Triton provides a flexible and scalable solution for deploying deep learning models, enabling organizations to efficiently deploy AI applications across a variety of hardware platforms, including GPUs and CPUs.

After setting up our NVIDIA Triton integration, we give you a dashboard for your NVIDIA Triton metrics.

Install the infrastructure agent

To use the NVIDIA Triton integration, you need to also install the infrastructure agent on the same host. The infrastructure agent monitors the host itself, while the integration you'll install in the next step extends your monitoring with NVIDIA Triton-specific data.

Enable the NVIDIA Triton integration with `nri-prometheus`

The Triton server metrics are displayed at the URL http://localhost:8002/metrics.

Dica

For additional details on collecting Triton server metrics, please refer to the NVIDIA documentation

To set up the NVIDIA Triton integration, follow these steps:

Run this command to create a file named nri-prometheus-config.yml in the integrations directory:
bash
```
$touch /etc/newrelic-infra/integrations.d/nri-prometheus-config.yml
```
Add the following snippet to your nri-prometheus-config.yml file to enable the agent to capture NVIDIA Triton data:
```
integrations:
  - name: nri-prometheus
    config:
      # When standalone is set to false nri-prometheus requires an infrastructure agent to work and send data. Defaults to true
      standalone: false

      # When running with infrastructure agent emitters will have to include infra-sdk
      emitters: infra-sdk

      # The name of your cluster. It's important to match other New Relic products to relate the data.
      cluster_name: "YOUR_DESIRED_CLUSTER_NAME"

      targets:
        - description: NVIDIA Triton metrics list
          urls: ["http://localhost:8002/metrics"]

      #     tls_config:
      #     ca_file_path: "/etc/etcd/etcd-client-ca.crt"
      #     cert_file_path: "/etc/etcd/etcd-client.crt"
      #      key_file_path: "/etc/etcd/etcd-client.key"

      # Whether the integration should run in verbose mode or not. Defaults to false
      verbose: false

      # Whether the integration should run in audit mode or not. Defaults to false.
      # Audit mode logs the uncompressed data sent to New Relic. Use this to log all data sent.
      # It does not include verbose mode. This can lead to a high log volume, use with care
      audit: false

      # The HTTP client timeout when fetching data from endpoints. Defaults to 30s.
      # scrape_timeout: "30s"

      # Length in time to distribute the scraping from the endpoints
      scrape_duration: "5s"

      # Number of worker threads used for scraping targets.
      # For large clusters with many (&gt;400) endpoints, slowly increase until scrape
      # time falls between the desired `scrape_duration`.
      # Increasing this value too much will result in huge memory consumption if too
      # many metrics are being scraped.
      # Default: 4
      # worker_threads: 4

      # Whether the integration should skip TLS verification or not. Defaults to false
      insecure_skip_verify: true
    timeout: 10s
```

NVIDIA Triton logs configuration

To configure nvidia triton logs, follow the steps outlined below.

Run this Docker command to check the status of running containers:
bash
```
$sudo docker ps
```
Copy the container ID for the nvidia-triton container and execute this command:
bash
```
$sudo docker logs -f <container_id> &> /tmp/triton.log &
```
Afterwards, verify there is a log file named triton.log located in the /tmp/ directory.

Forwarding NVIDIA Triton logs to New Relic

You can use our log forwarding to forward NVIDIA Triton logs to New Relic. On Linux machines, your log file named logging.yml should be in this path:

bash

$cd /etc/newrelic-infra/logging.d/

Once you find the log file in the above path, include this script into the logging.yml file:

logs:
  - name: triton.log
    file: /tmp/triton.log
    attributes:
      logtype: triton_logs

Restart the New Relic infrastructure agent

Run this command to restart your infrastructure agent:

bash

$sudo systemctl restart newrelic-infra.service

In a couple of minutes, your NVIDIA Triton server will send metrics to one.newrelic.com.

Find your data

You can choose our pre-built dashboard template named NVIDIA Triton to monitor your NVIDIA Triton server metrics. Follow these steps to use our pre-built dashboard template:

Go to one.newrelic.com > Integrations & Agents and type NVIDIA Triton.
Under Dashboards, click NVIDIA Triton.
Click Edit if you want to change the account in the open popup window.
Click Setup NVIDIA Triton or Skip this step if you already setup this datasource.
Click View dashboard, and see your NVIDIA Triton data in New Relic.
You can find your custom NVIDIA Triton dashboard in the Dashboards UI. See our dashboard section for more information.
Here is a NRQL query to check the NVIDIA Triton CPU memory:
```
SELECT latest(nv_cpu_memory_total_bytes) / 1e+6 AS 'memory (MB)' FROM Metric
```

What's next?

To learn more about building NRQL queries and generating dashboards, check out these docs:

Introduction to the query builder to create basic and advanced queries.
Introduction to dashboards to customize your dashboard and carry out different actions.
Manage your dashboard to adjust kkukkyour dashboards display mode, or to add more content to your dashboard.

Install the infrastructure agent .css-21sua1{background:none;border:none;width:0;padding:0;}

Enable the NVIDIA Triton integration with nri-prometheus