Muting rules: Suppress notifications

Alerts sends out timely notifications when your system is having problems. Sometimes you don't want to see some known notifications. You can use muting rules to stop being bombarded by messages that don't need your attention.

Once you've spotted the common elements in your unwanted notifications, you can define muting rules that specifically target those elements, while letting other notifications through. Even when a notification is muted, alertas still gathers data on those incidents. Muting rules don't interfere with the alerts process and are applied at the point right before a notification is sent.

Create a muting rule

To create a muting rule, follow these steps:

1. Go to one.newrelic.com > All capabilities > Alerts and click Muting rules on the left navigation pane.

2. Click + Add a rule.

3. Enter a name and a description (optional) for the muting rule, and select the account to which the rule will apply.

4. Build the incidents filter. You can use a subset of incident event attributes. Choose an attribute, an operator, and a value. These are the attributes: accountId, conditionId, conditionName, conditionType, entity.guid, nrqlEventType, nrqlQuery, policyId, policyName, product,runbookUrl (as conditionRunbookUrl), tags.<NAME>, and targetName). Values are compared against one of your incident attributes, such as an alerts policy ID or a condition name.

5. Click Add another condition if you want to include more filters.

Manage muting rules

A muting rule condition is the set of individual expressions made up of attributes, operators, and values that define which incidents to target for muting.

To create, enable, disable, and manage muting rules, follow these steps:

1. Go to one.newrelic.com > All capabilities > Alerts and click Muting rules on the left navigation pane.

2. Enable or disable muting rules at any time from the Enabled column. You also can edit each rule by clicking the icon on the row of each rule.

Rules can have one of the following statuses:

• Active: Muting is enabled and active.
• Scheduled: Muting is enabled but not active yet (there's a future schedule).
• Ended: Muting is enabled, but no longer active (there's no future schedule).
• Inactive: Muting is disabled.

Notification options for muting rules

When a muting rule is active, and an incident is opened, a user will not receive a notification. You can configure the behavior for notifications when a muting rule is inactive with the two settings below:

• Notify: If there’s an ongoing incident after the muting rule window ends, you will be notified. This works by closing the existing, muted incident, and if the threshold is still being violated, a new incident will open in a non-muted state, triggering a notification. We recommend keeping this default setting.

• Suppress notification: If there’s an ongoing incident after the muting rule window ends, you will not be notified. This works by leaving the existing, muted incident open past the muting rule window end timestamp.

Schedule a muting rule

If needed, you can schedule your muting rules.

To do this, select a start time and end time. Optionally, you can set the muting rule to last an entire day.

You can also choose to select a time zone for the muting rule schedule. The default is the time zone selected in your user preferences.

You can schedule your muting rules to recur daily, weekly, or monthly. A muting rule that's scheduled to repeat weekly includes the option to select the days of the week to recur. If no days are selected, the weekly recurrence will default to repeating on the day of the week that the muting rule is scheduled to start.

You can also specify when you would like recurrence to end by selecting either a specific date or a certain number of occurrences.

View muted incidents and issues

When viewing an open or closed issue, incidents and issues are marked as Muted. The following sections show some of these muted incidents and issues, and where you can find them.

An issue with a critical incident that's been muted

A list of incidents, some of which are muted

Incidents and issues are marked with the icon in the Muted column:

Mute faceted results using tags.

To mute results of faceted queries, use the tags.FACETED_ATTRIBUTE attribute, where FACETED_ATTRIBUTE represents the attributed you've run a NRQL FACET query on. For example: if your NRQL alert condition includes FACET host in its query, you can target that FACET attribute using tags.host.

NRQL condition queries can accept multiple facet attributes. If you want to be able to filter from attributes in your events or metric time series that have been aggregated, you must add those attributes to your NRQL query FACET clause; for example: FACET host, region, cluster.

For an example of using tags., see Create muting rule.

Sub-condition operators

These are the logical operators you can use to compare attributes when you're adding muting rules. If you're new to muting rules, see these examples.

• EQUALS: Where the supplied value equals the incident attribute value.
• DOES_NOT_EQUALS: Where the supplied value doesn't equal the incident attribute value.
• IN: Where the incident attribute value is present in a list of supplied values (up to 500).
• NOT_IN: Where the incident attribute value isn't present in a list of supplied values (up to 500).
• CONTAINS: Where the supplied value string is present in the incident attribute value.
• DOES_NOT_CONTAINS: Where the supplied value string isn't present in the incident attribute value.
• ENDS_WITH: Where the incident attribute value ends with the supplied value string.
• NOT_ENDS_WITH: Where the incident attribute value doesn't end with the supplied value string.
• STARTS_WITH: Where the incident attribute value begins with the supplied value string.
• DOES_NOT_STARTS_WITH: Where the incident attribute value doesn't begin with the supplied value string.
• IS_BLANK: Where the incident attribute value is blank. Null, empty string, etc.
• IS_NOT_BLANK: Where the incident attribute value is not blank. Null, empty string, etc.
• IS_ANY: A condition with this operator will mute all incidents on the account.

How muting rules work

Muting rules are applied at the end of the default alert lifecycle in order to suppress or mute notifications. They don't disable existing policies or conditions. For example, you can mute notifications during known system disruptions, such as maintenance windows and deployments. System disruption incidents will still be identified, even though the notifications for those incidents are muted.

A muting rule uses a set of conditions that match against attributes in an incident event. The muting rules tell us how to:

• Identify individual incidents after they're created, but before an issue is opened.
• Override their default condition to indicate that they should be muted.

Currently, muting an incident means that the normal alerting incident lifecycle is maintained, except that an issue containing only muted incidents will not send any notifications.

Muting rules are determined by the first event that triggered a notification within an issue. This means that if the first notification event was muted due to a muted state, the rest of the issue will be muted as well.

Muting rules override specific incidents. They don't disable existing policies or conditions. This allows you to mute incidents from specific entities that may be covered by a policy or condition that covers a large number of entities. This also keeps you from having to over-mute your monitoring when you are performing maintenance on a subset of your system.

The following table describes how the alerts incident lifecycle is affected by muted incidents:

Muting behavior with workflows

A triggered incident has a 1:1 ratio with an issue so if an incident is muted then the matching issue will be muted as well. Workflows are triggered by issues that can have one or more incidents, therefore there could be a scenario of muted and not muted incidents combined.

Each issue has one of the following muting states:

• Fully muted (FULLY_MUTED): an issue has all of its open incidents muted (Default value).
• Partially muted (PARTIALLY_MUTED): an issue that has at least one open incident that is muted and one open incident that is not muted.
• Not muted (NOT_MUTED): an issue that has no open muted incidents.

For a step-by-step guide on how to set up your workflows, check out an example demo below (approx. 2:17 minutes):

Muting behavior with NerdGraph

In NerdGraph, you can use the following queries and mutations with your muting rules. You can see the schema in more detail in the API Explorer.

• actor.account.alerts.mutingRule: Fetch a muting rule by ID.
• actor.account.alerts.mutingRules: Fetch a list of muting rules for an account.
• alertsMutingRuleCreate: Create a muting rule for an account.
• alertsMutingRuleUpdate: Update a muting rule by ID and account ID.

You can find some sample queries and mutations examples in this page.

A muting rule has the following fields and components:

Muting examples

For more information about making requests to NerdGraph, see the NerdGraph documentation, including GraphQL tutorials.

{
  actor {
    account(id: YOUR_ACCOUNT_ID) {
      alerts {
        mutingRules {
          name
          condition {
            operator
            conditions {
              attribute
              operator
              values
            }
          }
        }
      }
    }
  }
}

mutation {
  alertsMutingRuleCreate(
    accountId: YOUR_ACCOUNT_ID
    rule: {
      name: "host-1 mute rule"
      description: "Mute host-1 incidents"
      enabled: true
      condition: {
        operator: AND
        conditions: [
          { attribute: "tags.host", operator: EQUALS, values: ["host-1"] }
        ]
      }
    }
  ) {
    id
  }
}

mutation {
  alertsMutingRuleCreate(
    accountId: YOUR_ACCOUNT_ID
    rule: {
      name: "host-1 and synthetics/mobile mute rule"
      description: "Mute host-1 and synthetics/mobile incidents"
      enabled: true
      condition: {
        operator: OR
        conditions: [
          { attribute: "targetName", operator: EQUALS, values: ["host-1"] }
          {
            attribute: "product"
            operator: IN
            values: ["SYNTHETICS", "MOBILE"]
          }
        ]
      }
    }
  ) {
    id
  }
}

mutation {
  alertsMutingRuleUpdate(
    accountId: YOUR_ACCOUNT_ID
    id: MUTING_RULE_ID
    rule: { name: "updated name", enabled: false }
  ) {
    name
  }
}

mutation {
  alertsMutingRuleDelete(accountId: YOUR_ACCOUNT_ID, id: MUTING_RULE_ID) {
    id
  }
}