Manage your log collector lifecycle

Overview

The log collector lifecycle management solution in New Relic enables management of your log collection infrastructure across diverse environments. With this solution, you can configure and monitor the lifecycle of your log collectors.

You now have the ability to manage the entire lifecycle of your log collectors. This includes monitoring their health and configuring or deploying changes as needed.

This solution is useful for DevOps teams, system administrators, and IT operations managers who require visibility over their log collection infrastructure. You have access to tools that enhance your observability capabilities and maintain performance in both containerized and on-premise environments.

Solutions for diverse needs

• Log collector health monitoring: The Fluent Bit entity UI provides monitoring capabilities for log collectors.

• Troubleshooting: You can troubleshoot and fix configuration issues with your log collectors using the Fluent Bit entity UI, which helps identify and resolve problems.

• Configuration and remote deployment: With New Relic Control, you can centrally manage Fluent Bit configurations, update them, and deploy changes remotely.

  

Manage your Fluent Bit collector for Kubernetes clusters

The log collector lifecycle management solution currently provides support for Kubernetes environments. It enables you to:

• Gather and process logs from all your containerized applications.
• Append Kubernetes-related metadata to your logs.
• Use Fluent Bit's lightweight design to process large volumes of data without consuming a lot of resources.

Prerequisites

Before you install and configure Fluent Bit for logging with New Relic, ensure you have the following prerequisites:

• Access to a Kubernetes cluster
• The necessary tools and permissions for installation and configuration
• Network settings and security policies to allow communication with New Relic endpoints
• A New Relic account

For more information on these requirements, refer to Forward your logs to New Relic.
For information on New Relic Control setup, see Getting started with New Relic Control.

Set up

To install the Fluent Bit agent in Kubernetes clusters:

• Install the New Relic Kubernetes integration. This integration includes the Kubernetes plugin for logs.
• Use the configuration options available in the newrelic-logging repository during the guided install.

For details, refer to Enable Kubernetes for log management.

Install Fluent Bit with Fleet Control

If you're starting with Fleet Control, you'll set up your agent control configuration first. This process includes installing the Fluent Bit agent and other essential components.

To use Fluent Bit with Agent Control, enable log forwarding as part of the Agent Control installation, as shown in the sample configuration:

global:
cluster: "YOUR_CLUSTER_NAME"
licenseKey: "YOUR_LICENSE_KEY"
userKey: "YOUR_USER_KEY"

# Values related to the agent control's Helm chart release.
# `https://github.com/newrelic/helm-charts/blob/master/charts/agent-control/values.yaml`
agent-control-deployment:
  identityClientId: ""
  identityClientSecret: ""
  config:
      fleet_control:
      # Optional: Specify a fleet_id (entity guid) to automatically connect to an existing fleet.
      fleet_id: ""
      auth:
          # New Relic organization ID where agent will connect to.
          organizationId: "YOUR_ORGANIZATION_ID"
      
      # List of managed agents that will be deployed. The key represents the name of the agent and the value holds the configuration.
  subAgents:
    infrastructure:
      type: newrelic/com.newrelic.infrastructure:0.1.0
      content:
        # Ref: `https://github.com/newrelic/helm-charts/tree/master/charts/nri-bundle`
        # Recommended: check and define an explicit chart version (latest stable)
        chart_version: "*"   
        # chart_values:
        #   newrelic-infrastructure:
        #   enableProcessMetrics: true
    logs:
        type: newrelic/io.fluentbit:0.1.0
        content:
        # Ref: `https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging`
        # Recommended: check and define an explicit chart version (latest stable)
        chart_version: "*"
        chart_values:
            newrelic-logging:
              sendMetrics: true
  agent-operator:
    type: com.newrelic.k8s_agent_operator:0.1.0
    content:
      chart_version: "*"

Remember to uncomment the necessary configurations and replace placeholders such as YOUR_CLUSTER_NAME and YOUR_LICENSE_KEY with your actual values.

Manage and deploy configuration via Fleet Control

To add a new or edit an existing Fluent Bit configuration, use Fleet Control. The high-level tasks to manage and deploy a configuration are:

Create a configuration

To create a Fluent Bit configuration, follow these steps:

1. In New Relic, go to All Capabilities, search for New Relic Control, and then pin it to the side.
2. In New Relic Control, open Configurations.
3. Click Create configuration.
4. To create a configuration from scratch, click New in the drop-down menu. If you have an existing Fluent Bit configuration and want to leverage it, clone it by clicking Clone.
5. In the Create configuration dialog box, ensure Kubernetes is selected as the environment, select Fluent Bit as the agent type, and enter a name for the configuration. Then, click Continue.
6. In the Configuration column, copy the configuration template and then customize it per your requirements. For details, see supported parameters.
7. Save your changes. Your configuration will be added to the list of configurations on the Configuration page.
8. To edit your configuration, select the three dots (…) next to it, click Edit configuration, and make your changes. Then, click Save.

Create a deployment

1. In New Relic Control, open Fleets.
2. Select the fleet you want to create a Fluent Bit configuration for.
3. Click Create a deployment.
4. On the New deployment page, enter a deployment name and description, and then click Add a configuration.
5. Select Fluent Bit as the agent type, select the configuration you want to deploy, and then click Next.
6. Select the configuration version.

   Important

   A configuration can have multiple versions. Choose the version you want to deploy. For a newly created configuration, you’ll have only one version.
7. Click Add to deployment.

   Important

   Ensure the fleet has at least one managed entity. Or, add one.
8. Click Start deployment.

View fleet

In Fleets, select the specific fleet you want to work with. This opens the fleet detail page. The Summary tab shows the entities that are part of your fleet. To view agent or deployment details, use the Agents or the Deployments tab.

Monitor the health and performance of your log collector

Effective performance monitoring of the Fluent Bit agent is crucial for log management and optimization within your Kubernetes environments. By tracking key metrics, you can identify potential issues and facilitate troubleshooting.

Key performance metrics to track

• Log throughput rate: Measures the efficiency of log flow and highlights any disruptions. It includes an overall throughput assessment, which is further broken down by input and output plugins to show the flow of records and bytes processed.
• Memory buffer utilization: Helps identify resource constraints affecting log processing. It includes buffer usage, instances of overflows, queue length, and overall process and container memory utilization.
• Container events and crash loop backoffs: Alerts to abnormal conditions or failures in log pods, including errors, dropped records, and retries. These insights help in identifying and addressing issues that may disrupt log processing.

You can monitor the health and performance of your Fluent Bit agent on the Fluent Bit entity dashboard. Here’s how you can access it:

Via Integrations & Agents

1. Open the Installed tab. You’ll see the list of agents and integrations currently installed.
2. Use the search entities field or the Type filter to narrow down the list to Fluent Bit agents.
3. Select the agent that you want to check.

Via All Entities

1. Find the agent that you want to monitor under the Fluent Bit - Kubernetes group.
2. Select the agent.

Via Fleet

1. In New Relic Control, open Fleets.
2. Identify and select the fleet that has the Fluent Bit instrumentation.
3. On the Agents tab, select the Fluent Bit agent.

Example Use case: Log flow efficiency

Imagine that the logs in your Kubernetes cluster are not arriving at New Relic at the usual rate. Your typical log flow might be around 150,000 logs per minute, but current observations suggest a noticeable drop. Here’s how you can address this situation, leveraging New Relic Control and Fluent Bit monitoring tools to diagnose and resolve the issue.

Step 1: Monitor log throughput

Begin by checking the rate at which logs are forwarded. Discrepancies can indicate disruptions in the log forwarding process.

Step 2: Analyze container metrics and events

Access the Kubernetes overview to identify any abnormal events or restarts in the logging pods. Frequent restarts or warning events, such as crash loop backoffs, can disrupt your log pipeline and indicate underlying issues.

Step 3: Check memory usage

Examine the memory consumption of your Fluent Bit agents. High or erratic memory usage can signal configuration errors or resource limitations that need addressing.

Step 4: Utilize Fleet Control for troubleshooting

Use Fleet Control to review recent configuration changes. Identifying and rectifying any erroneous settings, such as incorrect memory limits, can resolve bottlenecks and improve log throughput.

Step 5: Redeploy and verify

After making necessary configuration adjustments in Fleet Control, initiate a new deployment to apply changes. Use the Fluent Bit entity view to monitor real-time metrics and observe the impact of the new configurations. Look for recovery in log throughput and stability in memory usage, indicating that the issues have been resolved.

Upgrade your Fluent Bit agent

Upgrade your Fluent Bit agent regularly to take advantage of the latest features, performance improvements, and security patches. You can learn directly from the New Relic UI when a newer version becomes available.

Before upgrading, ensure your current configurations are backed up. Review any release notes or documentation to understand new features or changes.

Steps to check if you are up-to-date

1. In your New Relic account, click Integrations & Agents, and then open the Installed tab.
   You’ll see the list of agents and integrations currently installed.
2. Use the search entities field or the Type filter to narrow down the list to Fluent Bit agents.
3. Identify the agent that requires an upgrade.
4. Click the Update from vx.y.x to X.Y.Z.
5. In the side panel that slides in, click View release notes.
6. On the Fluent Bit release notes page, note the latest Helm chart version.
7. Back in New Relic, close the side panel and click the fleet that contains the agent. It opens the fleet summary in New Relic Control.
8. Select the Fluent Bit configuration.
9. Update the chart version in the configuration file per the chart version you noted earlier, and save your changes.
10. Deploy the configuration.

After upgrading, verify log collection and processing to confirm the agent operates as expected. Use New Relic's monitoring tools to check that everything is running.