Monitor services running on Kubernetes

If you have services that run on Kubernetes and are supported by our applicable integrations, you can enable those integrations via our Kubernetes integration.

Get started

Our Kubernetes integration comes bundled with some of our on-host integrations. This lets you get data for those services by adding a section to the Kubernetes integration's configuration, which lives as a ConfigMap inside a manifest.

For an example of how to monitor Redis running on a Kubernetes PHP Guestbook, see this tutorial.

Requirements

To monitor services running on Kubernetes, you need:

• A Kubernetes cluster running a supported version.
• A Kubernetes cluster running the Kubernetes integration (install | check version | update).
• A supported service running on Kubernetes that meets our requirements. Supported services are:
  • Apache
  • Cassandra
  • Couchbase
  • Elasticsearch
  • HAProxy
  • HashiCorp Consul
  • JMX
  • Kafka
  • Memcached
  • MongoDB
  • MySQL
  • NGINX
  • PostgreSQL
  • RabbitMQ
  • Redis

For this install method, our RabbitMQ and Apache integrations do not report inventory data.

Enable monitoring of services using the Helm Chart

Monitoring services in Kubernetes works by leveraging our infrastructure agent and on-host integrations and an autodiscovery mechanism to point them to pods with a specified selector.

See this example of configuration for version 2.

To enable our Kubernetes integration to monitor one or more services:

Get the base configuration

You can get an example configuration file for our integrations that can run in K8s from the following links:

Please take into account that most of these examples will need to be configured for your particular environment, mainly to input the credentials required to authenticate against the particular service. You can read more about how to configure each integration in detail from the integration-specific pages, linked in the dropdown above.

Add the configuration to your values-newrelic.yaml

A working configuration snippet should be a YAML document with the following structure:

# Top level name can be arbitrary, akin to a config file name
redis-sampleapp.yaml:
  # Discovery section will define which pods will be monitored.
  discovery:
    command:
      # nri-discovery-kubernetes is a tool that connects to the Kubelet to fetch local pods
      # without putting stress in the API Server. It accepts the following arguments:
      # --namespaces: Comma separated namespaces to limit discovery on
      # --tls: Use tls for connecting to the kubelet
      # --port: Port used to connect to the kubelet. Defaults to 10255.
      exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250
      # Monitor pods which have a `app=sampleapp` label
      match:
        label.app: sampleapp

  # Integrations section contains the integration configs.
  # ${placeholders} will be replaced with the specified attributes for each pod that is discovered
  integrations:
    - name: nri-redis  # Integration name, should not be changed
      env:
        # Using the discovered pod IP as the hostname address
        HOSTNAME: ${discovery.ip}
        PORT: 6379
        # Other integration options go here

This snippet must be added to the integrations section, under newrelic-infrastructure in your values-newrelic.yaml file. For example:

global:
  licenseKey: _YOUR_NEW_RELIC_LICENSE_KEY_
  cluster: _K8S_CLUSTER_NAME_

# Other settings...

newrelic-infrastructure:
  # verboseLog: true
  integrations:
    redis-sampleapp.yaml:
      discovery:
        command:
          # --namespaces: Comma separated list of namespaces to discover pods on
          # --port: Port used to connect to the kubelet. Default is 10255
          # --tls: Use secure (TLS) connection
          exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250
          match:
            label.app: sampleapp

      integrations:
        - name: nri-redis
          env:
            HOSTNAME: ${discovery.ip}
            PORT: 6379

Integrations targeting other pods should have their own section next to redis-sampleapp.yaml.

Integrations are standalone binaries and are executed by the infrastructure agent that's included in the newrelic-nrk8s-kubelet-xxxxx pods. Config files are deployed to all pods of the nrk8s-kubelet DaemonSet, but discovery ensures that each pod will only target service pods that're scheduled in the same node as that particular nrk8s-kubelet pod. If an instance of the nrk8s-kubelet DaemonSet does not have any pods matching the specified labels, the integration will not be executed by that ### instance.

Verify the integration is working

Go to one.newrelic.com > All capabilities > Infrastructure, select Third party services, and then select the service's dashboard. You should see data being reported.

If you do not see the data there, the integration might be missing some parameter that it requires to run, or might not be able to reach the target service. You can get the logs of the integration by running:

$
kubectl logs -n newrelic newrelic-nrk8s-kubelet-xxxxx agent

Make sure to select the particular pod of the nrk8s-kubelet DaemonSet that is scheduled next to the pod that the integration should be targeting. You can check which instance is running on which node by running the following command:

$
kubectl get pods -n newrelic -o wide -l app.kubernetes.io/component=kubelet

Additional notes about enabling services

• Enabling multiple services may use more resources than what is set in the resource limits of the Kubernetes integration config file. If this becomes an issue, raise the limit in the resources section.
• The Kubernetes integration does not automatically update. For best results, regularly update.

Learn more

More resources for learning about configuration:

• Learn technical details about how configuration works.
• Learn how to configure monitoring of multiple services with the same config file.
• See a step-by-step tutorial showing how to monitor a Redis service on Kubernetes.

Enable monitoring of services using manifests

For each service you wish to monitor, you must add a configuration file for that integration to our Kubernetes integration's configuration. This document will cover these subjects:

• How the service-specific configuration YAML snippet works
• Adding the service-specific YAML in the Kubernetes integration's config file
• Adding multiple services to the Kubernetes integration's config file

How the service-specific YAML config works

Our Kubernetes integration's configuration follows the ConfigMap format. Using a ConfigMap allows us to decouple the configuration for the integrations from the Kubernetes image. The other benefit is that a ConfigMap can be updated automatically without reloading the running container.

Because the infrastructure agent uses YAML to configure its associated integrations, ConfigMaps are a good choice for storing YAML. (For more information on config file format, see the Integration config file format.)

The Kubernetes integration image comes with an auto-discovery feature that simplifies the configuration of multiple instances of services using a single configuration file. For example, if you have several NGINX instances running, creating an NGINX integration configuration file for every instance would be hard to implement and hard to update. With our auto-discovery option, you can discover and monitor all your NGINX instances with a single configuration file.

Each integration has its own specific configuration YAML. Our NGINX integration default config file looks like this:

discovery:
  command:
    # Use the following optional arguments:
    # --namespaces: Comma separated list of namespaces to discover pods on
    # --port: Port used to connect to the kubelet. Default is 10255
    # --tls: Use secure (TLS) connection
    # Custom Example:
    # exec: /var/db/newrelic-infra/nri-discovery-kubernetes --namespaces namespace1,namespace2 --port 10250 --tls
    # Default
    exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250
    match:
      label.app: nginx
integrations:
  - name: nri-nginx
    env:
      STATUS_URL: http://${discovery.ip}/status
      STATUS_MODULE: discover
      METRICS: 1

The above config enables the following:

• Runs nri-discovery-kubernetes to query the data for the node we are currently on.
• Parses the data that comes back and looks for any Kubernetes pod that has a Kubernetes container with an app= label with value nginx.
• For any matches, it attempts to run the NGINX integration. The status URL is built from:
  • The pod's IP address
  • The status page is pulled from the label on K8s pod called status_url

This automatic discovery works the same as the container auto-discovery used by the infrastructure agent. For more advanced options, see Container auto-discovery.

Add a service YAML to the Kubernetes integration config

The integration will recognize the above snippet, if you place it inside a designated ConfigMap. If you're using our Kubernetes integration v3, a ConfigMap with a name ending in -integrations-cfg should already exist.

If you're using Kubernetes integration version 2, see Add a service YAML.

Locate the config map and add the modified snippet to it, so it ends up looking like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: newrelic-infrastructure-integrations-cfg
  namespace: newrelic
data:
  nginx-config.yml: |
    discovery:
      command:
        # Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --port: Port used to connect to the kubelet. Default is 10255
        # --tls: Use secure (TLS) connection
        # Custom Example:
        # exec: /var/db/newrelic-infra/nri-discovery-kubernetes --namespaces namespace1,namespace2 --port 10250 --tls
        # Default
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          STATUS_URL: http://${discovery.ip}/status
          STATUS_MODULE: discover
          METRICS: 1

If you're using our Kubernetes integration v3, this ConfigMap will already be mounted in the required container.

Please note that the same ConfigMap can hold multiple config files, which is recommended to keep modifications to the manifests at a minimum.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: newrelic-infrastructure-integrations-cfg
  namespace: newrelic
data:
  nginx-config.yml: |
    discovery: # ...
    integrations:
      - name: nri-nginx
        # ...
  redis-config.yml: |
    discovery: # ...
    integrations:
      - name: nri-redis
        # ...

How to use the reported data

You can learn more about how to find and use your Kubernetes data here and you can review our K8sServiceSample data schema here.