Troubleshoot private locations

If you encounter problems while setting up private locations for synthetic monitoring of New Relic, try these troubleshooting tips to identify common minion or job manager problems.

Job demand and consumption

Identify your current throughput and assess whether your CPM or job manager is able to keep up with the job load.

• Jobs demand throughput per minute:

  SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
  WHERE type != 'SIMPLE' FACET location SINCE 2 days ago

  Copiar

• Jobs per minute over time:

  SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
  WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIES

  Copiar

• Identify queue growth:

  SELECT derivative(checksPending, 1 minute) AS 'queue growth rate (per minute)' 
  FROM SyntheticsPrivateLocationStatus WHERE name = 'YOUR_PRIVATE_LOCATION' SINCE 2 days ago TIMESERIES

  Copiar

• Consumption rate:

  SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest 
  WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIES

  Copiar

• Clean up failing monitors that are consuming resources:

  SELECT 100*latest(minionJobsFailed)/latest(minionJobsReceived) AS 'job failure rate (%)' 
  FROM SyntheticsPrivateMinion FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

  Copiar

Memory usage troubleshooting

Recommended memory allocation per CPM:

• Docker: 2.5 GiB per cpu core for Docker CPM

• Kubernetes CPM:

  • 3 GiB per CPM heavy worker
  • 3 GiB per healthcheck pod
  • 1.6 GiB for minion pod

• Target 60% max memory utilization

Recommended memory allocation per synthetics job manager:

• Docker: 3.256 GiB per cpu core for Docker synthetics job manager

• Kubernetes synthetics job manager (values are configurable):

  • 1.6 GiB for job manager pod
  • 1.0 GiB for ping runtime pod
  • 2 GiB per Node.js API runtime pod
  • 3 GiB per Node.js browser runtime pod

• Target 60% max memory utilization

Helpful queries to identify current usage:

• Memory usage per location:

  SELECT latest(minionPhysicalMemoryUsedPercentage) FROM SyntheticsPrivateMinion
  FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

  Copiar

• Memory usage per hostname:

  SELECT latest(minionPhysicalMemoryUsedPercentage) FROM SyntheticsPrivateMinion 
  WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES MAX

  Copiar

CPU usage troubleshooting

Identify potential periods of high cpu demand:

• Overall CPU usage:

  SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion 
  FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX

  Copiar

• CPU usage for a specified location:

  SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion 
  WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES MAX

  Copiar

Network troubleshooting

Network access is red:

The private minion makes a check to New Relic to verify public network access. If public network access is not available, the Network Access icon on the private minion's Overview page will turn red, but the private minion will otherwise be fully functional. If this is the expected behavior, you can safely ignore this error.

You may also see log entries like this:

2016-01-21 21:47:09,401 [dw-22 - GET /private-location/status] c.n.s.m.util.PrivateMinionUtils WARN Unable to detect Public Network Access trying to fetch https://newrelic.com/synthetics: java.net.ConnectException: Connection refused

New Relic access is red:

The private minion must have access to https://synthetics-horde.nr-data.net in order to retrieve the list of jobs to run and publish results that you can view in your New Relic user interface. If your firewall rules don't permit direct access, you must configure proxy access. If necessary, add the DNS endpoint to your allow list.

Network IOPS:

Network IOPS can also be a good place to troubleshoot if using a cloud provider for your CPM. Both Docker and Kubernetes CPMs require a decent amount of write throughput. Analyze your average IOPS and determine the appropriate volume size needed for your disk operations per second.

Port conflicts running Docker containers on the host

If you're running more than one Docker private location container in the same host, you'll have port conflicts. To avoid this port contention, make sure you do the following when you start setting up job managers:

• Run job managers and CPMs on different hosts.
• Run each job manager on a separate host.
• Run each CPM on a different host.