Log patterns are the fastest way to discover value in log data without searching.
Log data is high volume telemetry with a low value per individual record. Searching can quickly lead to logs that provide a root cause explanation, but most data is repetitive and hard to contextualize when browsing. Patterns can make log data discoverable without spending a lot of time reading through low value data.
one.newrelic.com > All capabilities > Logs > Patterns: Log patterns help you spot suspicious spikes or drops in log messages without having to spend extra time searching.
Technical overview
Log pattern functionality applies machine learning to normalize and group log messages that are consistent in format but variable in content. These grouped messages can be sorted, making it easy to find the most frequent or rarest sets of logs in your environment.
Use patterns as the basis for alerts when the frequency of important data changes, or to configure drop rules to get rid of unnecessary repetitive data.
Log patterns use advanced clustering algorithms to group together similar log messages automatically. With patterns, you can:
- Orient more quickly through millions of logs.
- Reduce the time it takes to identify unusual behavior in your log estate.
- Monitor the frequency of known patterns over time to focus your energy on what matters, and exclude what's irrelevant.
Availability
If you see Patterns are turned off in your logs UI, click Configure patterns and enable it. If you don't see patterns within 30 minutes of enabling the feature, there may be a lack of data with a message attribute for the system to create a pattern from it.
Log patterns | Limitations and considerations |
|---|---|
Pricing | There is no separate pricing for log patterns. The only cost is for additional data generated and added to your log records. A pattern attribute will be added to all logs that match a pattern. Attributes also may be added when common values are discovered, such as GUIDs, IP addresses, URL, or email addresses. These attributes are automatically extracted from the log message as part of the pattern process. |
HITRUST accounts | The log patterns feature is not FedRAMP compliant. FedRAMP or other HITRUST accounts are not eligible to use patterns. |
Log pattern matching limits | We have a system of safety limits on memory and CPU resources when matching logs to patterns. These matching limits can have an impact on the percentage of logs that are able to be grouped into log patterns. However, log pattern matching is a "best effort" process: it is not an error if not all of your logs have the opportunity to be grouped by patterns. You will still get value from the grouping that is able to be done within resource limits. |
Get started
To start examining patterns:
- Go to one.newrelic.com > All capabilities > Logs, and use the account picker dropdown to select the target account where you want to explore patterns.
- In the left navigation of the logs UI, click Patterns.
The main log UI changes to show patterns that match the query in the query bar. To look at logs for a specific time period, click that point (or click and drag an area) on the chart, or use the time picker.
Explore log patterns
By default the log patterns UI first shows the most frequent occurrence of patterns. To sort to show the rarest patterns first, click the Count column. You can also use the query bar or attributes bar to filter your log patterns.
If you want to... | Do this... |
|---|---|
Understand the rate of change in patterns | Look at the line chart. The color-coded patterns correspond to the plot column in the table. You can toggle individual plot patterns to narrow your focus. |
See the individual log messages that match each pattern | Click pattern to expand the row and see a table of individual log records.
|
Group and filter patterns by their attributes | Use the query bar and time picker. As you apply different filters and time windows, the log patterns adjust to your new target data. |
Create an alert from a pattern | Add the pattern to the query bar and run the query. Then click Create alert condition in the left nav. |
Troubleshoot log messages that haven't been clustered into a pattern | Use the Logs with no pattern tab in the Log patterns UI. |
Clicking a specific log message will open the log message details panel you're familiar with from the logs UI.
Explore logs with no pattern
The Logs with no pattern tab groups all recent log messages in your account that were not clustered into a known pattern yet. These log messages don't represent any problem or flaw in the system; they have no pattern because they are too new to have been processed by the machine learning system. This makes them valuable to explore when you want to understand what has recently changed in your environment.
For example:
- Are any of these logs tied to a recent problem? This is a quick way to discover unique log data that is appearing for the first time in your environment.
- Does your log data have a new format? Sometimes the logs don't represent a problem, but a new format of log data that deviates from the data model you expect your applications to follow.
Catching these logs early gives you the opportunity to ask developers to correct any deviations in their log output. The more consistent people are in the way log data is generated, the easier it becomes to use logs across a diverse set of teams.
Masked attributes and wildcards
Parts of the log messages in patterns are classified as variables and are substituted by masked attributes. The masking process supports and improves the clustering phase by allowing the algorithm to ignore changing details and focus on the repetitive structure.
Masked attributes include:
date_timeipurluuid
Masked attributes are highlighted and are easy to identify, as shown in the following example.
Log patterns extract other less trivial variables that don't belong to any masked attribute. These variables are indicated as wildcards *.
Troubleshooting
Here are a few reasons why you have patterns enabled but do not see any pattern data. If you're sure none of these items are true, get help from support.newrelic.com.
- No data has arrived in the timeframe you're observing. Try expanding the time range you're viewing with the time picker.
- It's been less than 24 hours since patterns were enabled in the account. This means the ML model may not be generated for the account yet.
- None of the data coming in has a
messagefield. Patterns will only be generated for values in themessagefield of a log record. If your logs don't containmessage, there will be no data.
Put the platform to work with patterns
Patterns are a value that is enriched onto the existing log message as a new attribute named newrelic.logPattern. Anything you can do with logs generally can be done with log patterns, such as:
- Build your own dashboards with patterns, to monitor a specific pattern or group of patterns you care about.
- Create for patterns by adding NRQL alerts.
- Use anomaly alert conditions to detect anomalies in known log patterns.