Kubernetes APM auto-attach

The Kubernetes APM auto-attach, formerly known as the Kubernetes agent operator, streamlines full-stack observability for Kubernetes environments by automating APM instrumentation alongside Kubernetes agent deployment. By enabling auto instrumentation, developers no longer need to manually manage APM agents. The Kubernetes APM auto-attach will automatically install, upgrade and remove APM agents.

It currently supports Java, .NET, Node.js, Python, Ruby, and PHP.

How it works

The MutatingWebHook, upon installation, becomes involved in intercepting API requests for deploying pods onto nodes.
Reflecting the configurations specified, it mutates the pod specification to add a NR init container and environment variables.
Following the establishment of the pod, the New Relic APM Agent is seamlessly integrated into the application housed within it.

Diagram showing how APM agents are auto injected

Before you begin

Before installing the operator, check the following:

Helm: You must install it to use the charts. See the Helm documentation if you need help getting started.
Kubectl: You have to configure it to communicate with your cluster.

Installation

Depending on what you need, you can choose to install the Kubernetes APM auto-attach independently or together with our Kubernetes integrations.

We strongly recommend to install it together with the Kubernetes integration to take advantage of our entire full stack observability experience.

Bundle installation in addition to the Kubernetes integration (recommended)

The Kubernetes APM auto-attach chart is part of the nri-bundle chart, which manages the installation of all the components needed to enable a full Kubernetes observability.

Add the k8s-agents-operator.enabled=true parameter to your helm command or include it in the values.yaml file. See the Install the Kubernetes integration page for more information about using Helm or check out the nri-bundle chart.

See this sample of Helm commands using parameters:

bash

$helm repo add newrelic https://helm-charts.newrelic.com
$
$helm upgrade --install newrelic-bundle newrelic/nri-bundle \
>    --set global.licenseKey=YOUR_NEW_RELIC_INGEST_LICENSE_KEY \
>    --set global.cluster=CLUSTER_NAME \
>    --namespace=newrelic \
>    --set newrelic-infrastructure.privileged=true \
>    --set global.lowDataMode=true \
>    --set kube-state-metrics.enabled=true \
>    --set kubeEvents.enabled=true \
>    --set k8s-agents-operator.enabled=true \
>    --create-namespace

Standalone installation

To install the Kubernetes APM auto-attach with the default configuration, run these commands:

bash

$helm repo add k8s-agents-operator https://newrelic.github.io/k8s-agents-operator
$helm upgrade --install k8s-agents-operator k8s-agents-operator/k8s-agents-operator \
>    --namespace newrelic \
>    --create-namespace \
>    --set licenseKey=YOUR_NEW_RELIC_INGEST_LICENSE_KEY

For a complete list of configuration options, see the README chart.

Configure auto-instrumentation

After APM auto-attach is all set up in your cluster, the next step is just to roll out the configs required to get it operational. That involves having at least one instrumentation Custom Resource (CR) active in the cluster.

Here's what the instrumentation CR lets you map out:

Name of the instrumentation CR
Where it will apply the instrumentation CR (thanks to podLabelSelector and namespaceLabelSelector)
APM agent (one per CR)
APM agent version
APM config parameters (env vars)
License key (optional)

The manifest file needs to injected in the same namespace (newrelic by default) where you installed APM auto-attach.

bash

$kubectl apply -f ./values.yaml -n newrelic

How to use selectors

To know when the instrumentation CR is going to inject APM agents, we need to use selectors. There are 2 label selectors available that you can use together (they act as a logical AND (&&) operator) or by separate depending on your needs.

PodLabelSelector informs the APM Auto-attach which pods needs to be instrumented.
Example using matchLabel (select pods containing an specific tag and value):
```
...
  podLabelSelector:
    matchLabels:
      app.kubernetes.io/name: flask-hello-world
  ...
```
NameSpaceLabelSelector defines at the namespace level the auto-instrumented pods.
Example using matchExpressions (select namespace containing an specific tag and value):
```
...
  namespaceLabelSelector:
    matchExpressions:
      - key: "kubernetes.io/metadata.name"
        operator: "In"
        values: ["backend"]
  ...
```
TIP
Keep in mind that apply the kubernetes.io/metadata.name label is the same as selecting based on the namespace's name.

Both selectors support matchLabel and matchExpressions.

matchExpressions is a more expressive label selector in Kubernetes and supports set-based matching unlike the matchLabels, which you can only use for exact matching. You can use it with or without the matchLabels selector.

...
  selector:
    matchLabels:
      tier: frontend
    matchExpressions:
      - {key: name, operator: In, values: [payroll, web]}
      - {key: environment, operator: NotIn, values: [dev]}
  ...

You can add more expressions to the selector. As in the example, each expression must contain a key, an operator, and possibly (depending on the operator) a list of values. There are four valid operators:

In: Label's value must match one of the specified values.
NotIn: Label's value must not match any of the specified values.
Exists: Pod must include a label with the specified key (the value isn't important). When using this operator, you shouldn't specify the values field.
DoesNotExist: Pod must not include a label with the specified key. You must not specify the values property.

If you specify many expressions, all those expressions must evaluate to true for the selector to match a pod. If you specify both matchLabels and matchExpressions, all the labels must match and all the expressions must evaluate to true for the pod to match the selector.

APM agent

You've got to specify the APM agent and its version within the instrumentation CR. We recommend using the latest version to take advantage of the newest features available.

Language	Image	Available versions
dotnet	`newrelic-dotnet-init:latest`	.NET
java	`newrelic-java-init:latest`	Java
nodejs	`newrelic-node-init:latest`	Node
python	`newrelic-python-init:latest`	Python
ruby	`newrelic-ruby-init:latest`	Ruby
php	`newrelic-php-init:latest`	PHP

See this example:

...
  spec:
    agent:
      language: dotnet
      image: newrelic/newrelic-dotnet-init:latest
  ...

APM configuration parameters

The instrumentation CR provides the capability to inject environment variables in the pod to streamline the configuration of the APM agents. See this example:

...
  spec:
      env:
      # Example overriding the appName configuration by using a label of the pod
        - name: NEW_RELIC_APP_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['app.kubernetes.io/name']
  ...

In the above example, we show you how you can configure the agent settings globally using environment variables. See each agent's configuration documentation for available configuration options:

Important

You can inject these environment variables in the app deployment manifest.

License keys (optional)

When you install it, a is created and it's the license by default. Follow these steps, if you need to send the APM telemetry to a different account:

To create a secret containing a new license key, run this command:

bash

$kubectl create secret generic newrelic-key-secret \
>  --namespace my-monitored-namespace \
>  --from-literal=new_relic_license_key=<NEW RELIC INGEST LICENSE KEY>

To reference the secret from the instrumentation CR, run this command:
```
...
spec:
  licenseKeySecret: the-name-of-the-custom-secret
...
```

Instrumentation CR examples

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-python
  namespace: newrelic
spec:
  agent:
    language: python
    image: newrelic/newrelic-python-init:latest
    env:
      - name: NEW_RELIC_APP_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['app']
  podLabelSelector:
    matchExpressions:
      - key: "app"
        operator: "In"
        values: ["flask-hello-world","flask-hello-world-v2"]

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-java
  namespace: newrelic
spec:
  agent:
    language: java
    image: newrelic/newrelic-java-init:latest
  namespaceLabelSelector:
    matchExpressions:
      - key: "kubernetes.io/metadata.name"
        operator: "In"
        values: ["java"]

apiVersion: newrelic.com/v1alpha2
kind: Instrumentation
metadata:
  name: newrelic-instrumentation-ruby
  namespace: newrelic
spec:
  agent:
    language: ruby
    image: newrelic/newrelic-ruby-init:latest
  namespaceLabelSelector:
    matchExpressions:
      - key: "Ruby"
        operator: "Exists"
  licenseKeySecret: the-name-of-the-custom-secret

Update APM instrumentation in applications

By default, the Kubernetes APM auto-attach automatically installs the latest available version of the corresponding APM agent.

Once the monitoring of an application starts, it's not automatically updated to a newer version unless you choose to update. You can update the application by redeploying the pods or restarting your deployment.

Remove APM instrumentation in applications

To remove the APM instrumentation from an application, you must change the matching label selector inside either the podLabelSelector or namespaceLabelSelector used or delete the instrumentation CR. Then, restart the deployment. The remove process takes just a few seconds.

Update the Kubernetes APM auto-attach

Bundle installation

Run an update of the nri-bundle chart with the following parameter:

bash

$k8s-agents-operator.enabled=true

Standalone installation

Run the helm upgrade command to update to a newer version of the Kubernetes APM auto-attach.

bash

$helm upgrade k8s-agents-operator newrelic/k8s-agents-operator -n newrelic

Uninstalling the Kubernetes APM auto-attach

Bundle installation

Uninstall the nri-bundle chart or if you only want to remove Kubernetes APM auto-attach, run a helm upgrade with the following parameter:

bash

$k8s-agents-operator.enabled=false

Standalone installation

To uninstall and delete the Kubernetes APM auto-attach, run this command:

bash

$helm uninstall k8s-agents-operator -n newrelic

Find and use data

Get insights of your applications and resolve incidents using the APM summary page.
Check out the Kubernetes summary page. It provides Kubernetes insights in the context of your monitored applications.

Certificates

The Kubernetes APM auto-attach can support cert-manager if preferred.

Run this command to install the cert-manager Helm chart:

bash

$helm install cert-manager jetstack/cert-manager \
>  --namespace cert-manager \
>  --create-namespace \
>  --set crds.enabled=true

In your values.yaml file, set admissionWebhooks.autoGenerateCert.enabled: false and admissionWebhooks.certManager.enabled: true. Then, install the chart as normal.

Available chart releases

Run this command to see the available charts:

bash

$helm search repo k8s-agents-operator

Frequently asked questions

Yes, custom instrumentation will work the same as without APM auto-attach. The main difference is that the agent is now injected by APM auto-attach instead of installed in the container with the rest of the application dependencies.

You can still import and call the agent API to add custom instrumentation into your application. You can also utilize a configuration file or environment variables to add custom instrumentation if the particular agent you're using supports it. Note that agents have order of precendence between configuration via environment variables and configuration via configuration files, so you will need to make sure your environment variable configuration via the operator is not clashing with your configuration via configuration file. See each agents custom instrumentation docs for details:

Troubleshooting

If your applications are not instrumented, you should check the following:

Please be sure to redeploy or deploy new applications after you've installed Kubernetes APM auto-attach. Notice that only auto-instruments new applications are deployed in the cluster.
Run this command to check that the secret is installed in the app's namespace:
bash
```
$kubectl get secrets -n NAMESPACE
```
Check that the pod has the required labels that enable automatic instrumentation through CR when using podLabelSelector. Similarly, check that the namespace has the required labels when using namespaceLabelSelector inside the CR.
bash
```
$kubectl get pod POD_NAME -n NAMESPACE  -o jsonpath='{.metadata.annotations}'
```
Run this command to get logs from the APM auto-attach pod:
bash
```
$kubectl logs AGENT_OPERATOR_POD -n newrelic
```
Run this command to ensure the init container has been injected and sucessfully executed inside the application's pod.
bash
```
$kubectl describe pod POD_NAME -n NAMESPACE
```

How to migrate from previous versions that required annotations

Starting with version 0.14, annotations within the application deployment manifest are no longer necessary for applications to be auto-instrumented.

It's advised to uninstall any versions preceding 0.14 and proceed with the installation of the latest release. Utilizing the label selectors within the instrumentation CR will enable the precise deployment of APM agents, thereby obviating the requirement for annotations.

Support

Kubernetes APM auto-attach supports the following languages and their minimum supported versions according to our standard APM agent support policy:

Java: 8.12
.NET: 10.25
Ruby: 9.10
Node.js: 11.9
Python: 9.10
PHP: 11.12

For any issues:

Review the issues section on GitHub for any similar problems or consider opening a new issue.
You can reach out to the New Relic support team for assistance.

How it works

Before you begin

Installation

Bundle installation in addition to the Kubernetes integration (recommended)

Standalone installation

Configure auto-instrumentation

How to use selectors

TIP

How to use `matchLabel` and `matchExpressions`

APM agent

APM configuration parameters

Important

License keys (optional)

Instrumentation CR examples

Python agent: Pods with a specific label and value and overriding app name

Java agent: Pods of a specific namespace

Ruby agent: Any namespace containing the Ruby label and sending data to a different account

Update APM instrumentation in applications

Remove APM instrumentation in applications

Update the Kubernetes APM auto-attach

Bundle installation

Standalone installation

Uninstalling the Kubernetes APM auto-attach

Bundle installation

Standalone installation

Find and use data

Certificates

Available chart releases

Frequently asked questions

Can I route my applications telemetry to different accounts?

Important

Can I install the Kubernetes APM auto-attach if my applications are already instrumented?

Can I use custom instrumentation with the Kubernetes APM auto-attach?

Can I install the Kubernetes APM auto-attach if my applications are running on a read-only file system?

Can I setup Kubernetes APM auto-attach in Windows nodes?

Can I setup Kubernetes APM auto-attach in Fargate nodes?

Troubleshooting

How to migrate from previous versions that required annotations

Support

Kubernetes APM auto-attach

How it works .css-21sua1{background:none;border:none;width:0;padding:0;}

Before you begin

Installation

Bundle installation in addition to the Kubernetes integration (recommended)

Standalone installation

Configure auto-instrumentation

How to use selectors

TIP

APM agent

APM configuration parameters

Important

License keys (optional)

Instrumentation CR examples

Python agent: Pods with a specific label and value and overriding app name

Java agent: Pods of a specific namespace

Ruby agent: Any namespace containing the Ruby label and sending data to a different account

Update APM instrumentation in applications

Remove APM instrumentation in applications

Update the Kubernetes APM auto-attach

Bundle installation

Standalone installation

Uninstalling the Kubernetes APM auto-attach

Bundle installation

Standalone installation

Find and use data

Certificates

Available chart releases

Frequently asked questions

Can I route my applications telemetry to different accounts?

Can I install the Kubernetes APM auto-attach if my applications are already instrumented?

Can I use custom instrumentation with the Kubernetes APM auto-attach?

Can I install the Kubernetes APM auto-attach if my applications are running on a read-only file system?

Can I setup Kubernetes APM auto-attach in Windows nodes?

Can I setup Kubernetes APM auto-attach in Fargate nodes?

Troubleshooting

How to migrate from previous versions that required annotations

Support

How it works