OpenTelemetry APM UI

The OpenTelemetry APM UI provides a comprehensive monitoring experience for services instrumented with OpenTelemetry, offering the same powerful APM capabilities you expect from New Relic's traditional language agents.

Prerequisites

Before using the OpenTelemetry APM UI, ensure you have:

Configured your service with OpenTelemetry instrumentation
Set up your service to send data to New Relic
Created OpenTelemetry service entities

If you haven't completed these steps, see OpenTelemetry APM monitoring for setup instructions.

Find your OpenTelemetry services

To locate your OpenTelemetry-instrumented services:

In the New Relic UI, navigate to All entities > Services - OpenTelemetry or APM & Services
Select a service to open its Summary page

팁

Use entity tags in the entity explorer to filter services. Learn more about how entity tags are computed in OpenTelemetry resources in New Relic.

How OpenTelemetry works with New Relic APM

OpenTelemetry-instrumented services provide the same curated APM experience as services using New Relic's language agents. Here's how it works:

Data mapping process

New Relic automatically maps your OpenTelemetry data to our APM conventions by:

Generating APM metrics: We create the metrics needed for the APM experience directly from your OpenTelemetry data
Preserving original data: Your original OpenTelemetry data remains available for custom dashboards and alerts
Normalizing conventions: We handle OpenTelemetry's evolving semantic conventions so you don't have to track version differences

Benefits for existing New Relic users

If you're transitioning from New Relic agents to OpenTelemetry, you can continue using familiar metrics and queries while adopting OpenTelemetry standards.

중요

Why we normalize OpenTelemetry data

OpenTelemetry's semantic conventions are still evolving, with many not yet stable. By normalizing your data to New Relic conventions, we:

Reduce the complexity of tracking which OpenTelemetry convention versions your instrumentation uses
Provide a consistent experience as you transition from New Relic agents to OpenTelemetry
Ensure your APM experience remains stable regardless of OpenTelemetry changes

Data sources and prioritization

The APM experience uses three types of OpenTelemetry data:

Metrics (primary): Provide accurate service statistics like throughput, response time, and error rate
Spans (supplementary): Used when metrics aren't available or for specific features like transaction traces
Logs: Integrated for troubleshooting and correlation

Why metrics are preferred: Metrics give a complete picture of your service performance, while spans are typically sampled and may not represent all traffic.

The APM experience primarily leverages these OpenTelemetry semantic conventions:

HTTP - Web transactions and external calls
RPC - Remote procedure calls
Messaging - Message queue operations
Database - Database operations

How transactions are derived from OpenTelemetry data

New Relic’s APM experience is centered around the notion of a transaction. When using a New Relic agent, it is the responsibility of the agent’s instrumentation to define the scope of a transaction (e.g., a single web request). The agent produces metric data that drives the vast majority of the New Relic APM experience by recording transaction metrics measuring the duration of transactions and its individual operations (e.g., external calls and database calls).

OpenTelemetry instrumentation does not have a direct analog to a New Relic transaction, so adapting the notion of transactions to OpenTelemetry data is key.

By leveraging OpenTelemetry’s semantic conventions we can take advantage of OpenTelemetry’s highly structured and standardized means of describing telemetry in order to drive the APM experience in a very similar way to that of New Relic’s agents.

The semantic conventions define standard metrics for measuring common operations such as handling HTTP or RPC requests. These metrics are analogous to the transaction metrics that New Relic agents produce for describing Web transactions. We leverage OpenTelemetry’s HTTP and RPC metrics to synthesize metrics that drive the APM UI such as the apm.service.transaction.duration metric.

New Relic also provides a notion of a non-web transaction. Non-web transactions are commonly used for systems that perform message processing. Instrumentation that leverages OpenTelemetry’s messaging conventions will result in synthesizing metrics representing non-web transactions.

중요

Messaging operations and span data

OpenTelemetry's messaging conventions are less mature than HTTP and RPC conventions. Currently, we generate non-web transaction metrics for messaging operations from span data rather than metric data. This approach follows the messaging semantic conventions but may be affected by your sampling strategy.

Transaction names

Each transaction has a name which is derived from attributes required by OpenTelemetry’s semantic conventions. Refer to the APM Service Metrics section for how this name is derived.

Unknown transaction name

Sometimes you may see a transaction with unknown in the name. This is an indication that the source data used to derive the transaction does not follow any of the established OpenTelemetry semantic conventions which we currently support.

Some examples:

HTTP metrics missing http.request.method or http.route. For example, if the http.server.request.duration metric is missing the http.route attribute then the transaction name will be WebTransaction/server/GET unknown.
Frameworks or protocols for which OpenTelemetry does not currently define semantic conventions for (e.g., background jobs and CI frameworks).

Navigating the APM experience

Summary

The summary page provides an overview of your service's health and is centered around New Relic's notion of a transaction. See How transactions are derived from OpenTelemetry data for more details.

The New Relic metrics that drive the Summary page are the apm.service.transaction.duration and apm.service.error.count metrics. Refer to them for details on how they are derived from your OpenTelemetry data.

Customizing Apdex Target

In New Relic instrumentation, custom apdex targets are configured using agent configuration. For OpenTelemetry, when viewing a service navigate to Settings > Application to configure your Apdex target.

Distributed tracing

The distributed tracing page provides detailed insights into OpenTelemetry trace data. See Distributed tracing for page usage information. See OpenTelemetry traces in New Relic for details on how OpenTelemetry trace data is ingested into New Relic.

As with golden signals, spans are classified as errors if the span status is set to ERROR (for example, otel.status_code = ERROR). If the span is an error, the span status description (for example, otel.status_description) is displayed in error details.

OpenTelemetry span events attach additional event context information to a particular span. They are most commonly used to capture exception information. If available, you can view a span's events in trace details.

팁

The presence of a span exception event doesn't qualify the span as an error on its own. Only spans with span status set to ERROR are classified as errors.

Screenshot showing the right pane showing the two links for span events

Service map

The service map page provides a visual representation of your entire architecture. See Service maps for more information.

Transactions

The transactions page provides tools for identifying problems with and analyzing a service's transactions.

OpenTelemetry does not have a direct analog to New Relic's notion of a transaction. See How transactions are derived from OpenTelemetry data for more details.

The New Relic metrics that drive the Transaction page are the apm.service.transaction.duration and apm.service.error.count metrics. Refer to them for details on how they are derived from your OpenTelemetry data.

Transaction traces

Transaction traces for OpenTelemetry are derived from your span data. On the transactions page you can find a list of transaction traces. This list requires the span data and metric data for a given transaction be correlated together. We do this by adding a transaction.name attribute to the root span of a transaction trace.

Segment breakdown

Clicking on a transaction opens a detailed view of the transaction revealing a segment breakdown. Unlike New Relic agents, OpenTelemetry does not emit metrics for individual segments. Therefore, the New Relic metrics required for driving the segment breakdown are derived from span data.

The notable downside to computing segment breakdown from span data is that spans are usually sampled. Though, even with sampling, the segment breakdown can still serve to help you identify specific methods or operations that consume the most time within a transaction.

From span data, we estimate a sampling rate by computing a throughput based on span data received and divide that by the actual throughput as reported by your metric data. The estimated sampling rate enables us to extrapolate the segment breakdown of a transaction.

This process is not perfect and can be affected by a number of factors, most notably your sampling strategy. It works best when you're strictly sampling a percentage of your span data. However, for example, if you sample only spans representing errors, then the segment breakdown may be skewed.

Databases

The databases page provides tools for identifying problems with and analyzing a service's database client operations.

OpenTelemetry instrumentation represents calls to databases using the database semantic conventions.

The New Relic metric that drives the databases page is the apm.service.datastore.operation.duration metric. Refer to it for details on how it is derived from your OpenTelemetry data.

Time consumption by caller

When you click on a specific database call, you will see the "Time consumption by caller" chart. This chart is driven by the apm.service.transaction.overview metric. This is the same metric that drives the segment breakdown view of the transaction page and it is derived from span data.

중요

OpenTelemetry's database semantic conventions were recently deemed stable. There does not yet exist many stable instrumentations, and the instrumentation that does exist often only emits span data and no metric data.

As such, when using instrumentation that has not yet adopted the stable semantic conventions, the APM metrics generated driving the database page are derived from span data.

As stable instrumentation becomes available and you adopt it, the database page will begin to leverage OpenTelemetry metric data. Reach out to the OpenTelemetry community regarding the availability of stable database instrumentation in your language.

External services

The external services page provides tools for identifying problems with and analyzing a service's external calls, including calling entities (upstream services) and called entities (downstream services).

OpenTelemetry instrumentation represents calls to external services using the HTTP and RPC semantic conventions.

The New Relic metric that drives the databases page is the apm.service.external.host.duration metric. Refer to it for details on how it is derived from your OpenTelemetry data.

Time consumption by caller

When you click on a specific external call, you will see the "Time consumption by caller" chart. This chart is driven by the apm.service.transaction.overview metric. This is the same metric that drives the segment breakdown view of the transaction page and it is derived from span data.

JVM runtime

The JVM runtime page provides tools for identifying problems with and analyzing a Java service's JVM. The page is only displayed for services using OpenTelemetry Java. In order to differentiate between distinct service instances, the page requires the service.instance.id resource attribute to be set (see Services for more details).

The JVM runtime page shows golden signals alongside JVM runtime metrics to correlate runtime issues with service usage.

The queries assume data conforms to the JVM metric semantic conventions. Note, these conventions are embodied in OpenTelemetry Java runtime instrumentation library, which is automatically included with the OpenTelemetry java agent.

Go runtime

The Go runtime page provides tools for identifying problems with and analyzing a Go service's runtime. The page is only displayed for services using OpenTelemetry Go. In order to differentiate between distinct service instances, the page requires the service.instance.id resource attribute to be set (see Services for more details).

The Go runtime page shows golden signals alongside Go runtime metrics to correlate runtime issues with service usage.

The queries assume data is produced by the OpenTelemetry Go runtime instrumentation library. Note, there are currently no semantic conventions for Go runtime metrics.

Logs

The Logs page provides tools for identifying problems and analyzing a service's logs. See Use logs UI for more information.

Errors inbox

The errors inbox page provides tools for detecting and triaging a service's errors. See Getting started with errors inbox for more details.

The errors inbox page is driven by trace data. As with golden signals, spans are classified as errors if the span status is set to ERROR (for example, otel.status_code = ERROR).

Error spans are grouped together by their error fingerprint, computed by normalizing away identifying values such as UUIDs, hex values, email addresses, etc. Each distinct error span is an individual instance within the error group. The error group message is determined as follows:

Span status description (for example, otel.status_description)
rpc.grpc.status_code from RPC span semantic conventions
http.status_code from HTTP span semantic conventions
http.response.status_code from HTTP span semantic conventions
undefined if none of the above are present

Span View (Legacy)

In the past, OpenTelemetry instrumented services lit up an entirely different user experience from that of New Relic's language agents. This older experience provided curated charts driven from span data. Span data is usually sampled, so it can be misleading particularly when representing things like throughput.

For now, the old user experience is still available via the Span View (Legacy) page. At the top, it contains four tabs: Summary, Transactions, Databases, and External services. All the charts on these tabs are driven from span data.

팁

Legacy span-based view

Our older OpenTelemetry APM experience allowed viewing data from both metric and span perspectives. Since span data is typically sampled, metrics provide more accurate throughput and response time measurements. The span-based view is still available but will be phased out. See Span View (Legacy) for details.

New Relic APM metrics derived from span data

The New Relic APM metrics that drive the APM experience are usually derived from metric data. However, there are a few scenarios where APM metrics are derived from span data. For the following scenarios note that the sampling strategy you're using will influnce the metrics generated.

Segment breakdown

The transaction segment breakdown view is driven from span data. See Segment breakdown for more information.

Database calls

The OpenTelemetry database semantic conventions were recently declared stable. However, most database instrumentation has not yet adopted the stable conventions and does not yet emit metric data. Therefore, when using older instrumentation, the metrics that drive the database page are generated from span data. You're encouraged to upgrade to the latest stable database instrumentation as soon as it becomes available for your language.

Messaging systems

The OpenTelemetry messaging semantic conventions are still in development. Most messaging instrumentation does not yet emit metric data. New Relic represents interations with messaging systems as non-web transactions. See How transactions are derived from OpenTelemetry data for more information.

OpenTelemetry Ruby

OpenTelemetry supports metrics for most languages now, however Ruby is a notable exception. For Ruby, we perform a best-faith effort to generate New Relic APM metrics from span data.

APM Service Metrics

The apm.service.* metrics drive New Relic’s APM experience. The following sections describe the source OpenTelemetry data that is used to derive apm.service.* metrics.

Metric resource attributes

The following resource attributes are copied from source data on to APM metrics:

container.id
entity.guid
host.name
instrumentation.provider
k8s.cluster.name
k8s.container.name
k8s.namespace.name
k8s.pod.name
service.instance.id
service.name

Metric: `apm.service.transaction.duration`

Name	Instrument Type	Unit (UCUM)	Description
`apm.service.transaction.duration`	Distribution	`s`, `ms`, `ns` [1]	Duration of transactions. [2]

Attribute	Type	Description	Example
`transactionName`	`string`	The name of the transaction.	`WebTransaction/server/GET /users/:id`, `OtherTransaction/consumer/unknown`
`transactionType`	`string`	The type of the transaction.	`Web`, `Other`
`error.type`	`string`	Describes a class of error the transaction ended with.	`500`, `TimeoutException`

[1]: Unit from source metric is copied.

[2]: If error.type resolves to non-null, apm.service.error.count is incremented with the respective count of the source data.

Metric sources

Semantic Convention	Metric name	Conditions	`transactionName`	`transactionType`	`error.type`
`OtelHttpServer1_23`	`http.server.request.duration`	`http.request.method IS NOT NULL`	`WebTransaction/server/${http.request.method} ${http.route}`	`Web`	`${error.type}`
`OtelHttpServer1_20`	`http.server.duration`	`http.method IS NOT NULL`	`WebTransaction/server/${http.method} ${http.route}`	`Web`	`__http_error_status_code_or_null__`
`OtelRpcServer1_20`	`rpc.server.duration`	`rpc.system IS NOT NULL`	`WebTransaction/server/${rpc.system}/${rpc.service:-unknown}.${rpc.method:-unknown}`	`Web`	`__rpc_error_status_code_or_null__`

Metric: `apm.service.transaction.overview`

Name	Instrument Type	Unit (UCUM)	Description
`apm.service.transaction.overview`	Summary	s	Segment breakdown time of a transaction.

Attribute	Type	Description	Example
`transactionName`	`string`	The name of the transaction.	`WebTransaction/server/GET /users/:id`, `OtherTransaction/consumer/unknown`
`transactionType`	`string`	The type of the transaction.	`Web`, `Other`
domain attributes	various	Domain specific attributes dependent on the source convention including: `db.system`, `db.sql.table`, `db.operation`, `external.host`	See `apm.service.external.host.duration`, `apm.service.datastore.operation.duration`

Span sources

Semantic Convention	Span kind	Conditions	`transactionName`	`transactionType`	domain attributes
`OtelHttpServer1_23`	`server`	`http.request.method IS NOT NULL`	`WebTransaction/server/${http.request.method} ${http.route}`	`Web`
`OtelHttpServer1_20`	`server`	`http.method IS NOT NULL`	`WebTransaction/server/${http.method} ${http.route}`	`Web`
`OtelRpcServer1_20`	`server`	`rpc.system IS NOT NULL`	`WebTransaction/server/${rpc.system}/${rpc.service:-unknown}.${rpc.method:-unknown}`	`Web`
`OtelMessagingConsumer1_24`	`consumer`	`messaging.operation IS NOT NULL`	`OtherTransaction/consumer/${messaging.operation:-unknown}/${messaging.destination.template:-${messaging.destination.name:-unknown}}`	`Other`
`OtelDbClient1_33`	`internal` `client`	`db.system.name IS NOT NULL`	`transactionName` of the local root span	`transactionType` of the local root span	`db.system`: `${db.system.name}` `db.sql.table`: `${db.stored_procedure.name:-${db.collection.name:-${__db_summary_to_sql_table__}}}` `db.operation`: `${db.operation.name:-${__db_summary_to_operation__:-unknown}}`
`OtelDbClientRedis1_24`	`client`	`db.system IS NOT NULL` `db.system = 'redis'`	`transactionName` of the local root span	`transactionType` of the local root span	`db.system`: `${db.system}` `db.sql.table`: `${db.sql.table}` `db.operation`: `${db.operation:-${name:-unknown}}`
`OtelDbClient1_24`	`client`	`db.system IS NOT NULL`	`transactionName` of the local root span	`transactionType` of the local root span	`db.system`: `${db.system}` `db.sql.table`: `${db.sql.table}` `db.operation`: `${db.operation:-unknown}`
`OtelHttpClient1_23`	`client`	`http.request.method IS NOT NULL`	`transactionName` of the local root span	`transactionType` of the local root span	`external.host`: `${server.address:-unknown}`
`OtelHttpClient1_20`	`client`	`http.method IS NOT NULL`	`transactionName` of the local root span	`transactionType` of the local root span	`external.host`: `${net.peer.name:-unknown}`
`OtelRpcClient1_20`	`client`	`rpc.system IS NOT NULL`	`transactionName` of the local root span	`transactionType` of the local root span	`external.host`: `${net.peer.name:-unknown}`

Metric: `apm.service.external.host.duration`

Name	Instrument Type	Unit	(UCUM)	Description
`apm.service.external.host.duration` \| \| \|	Distribution	`s`, `ms`, `ns` [1]	Duration of external calls.

Attribute	Type	Description	Example
`external.host`	`string`	Server domain if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name	`example.com`, `10.1.2.80`, `/tmp/my.sock`

[1]: Unit from source metric is copied.

Metric sources

Semantic Convention	Metric name	Conditions	`external.host`
`OtelHttpClient1_23`	`http.client.request.duration`	`http.request.method IS NOT NULL`	`${server.address:-unknown}`
`OtelHttpClient1_20`	`http.client.duration`	`http.method IS NOT NULL`	`${net.peer.name:-unknown}`
`OtelRpcClient1_20`	`rpc.client.duration`	`rpc.system IS NOT NULL`	`${net.peer.name:-unknown}`

Metric: `apm.service.datastore.operation.duration`

Name	Instrument Type	Unit (UCUM)	Description
`apm.service.datastore.operation.duration`	Distribution	`s`, `ms`, `ns` [1]	Duration of datastore calls.

[1]: Unit from source metric is copied.

Attribute	Type	Description	Example
`db.system`	`string`	The database management system (DBMS) product as identified by the client instrumentation.	`postgresql`, `mysql`, `mariadb`
`db.sql.table`	`string`	The name of a collection (table, container) within the database.	`public.users`, `customers`
`db.operation`	`string`	The name of the operation or command being executed.	`findAndModify`, `HMSET`, `SELECT`
`db.query.summary`	`string`	Low cardinality summary of a database query.	`SELECT wuser_table`, `INSERT shipping_details`, `SELECT order`

Metric sources

Semantic Convention	Metric name	Conditions	`db.system`	`db.sql.table`	`db.operation`	`db.query.summary`
`OtelDbClient1_33`	`db.client.operation.duration`	`db.system.name IS NOT NULL`	`${db.system.name}`	`${db.stored_procedure.name:-${db.collection.name:-${__db_summary_to_sql_table__}}}`	`${db.operation.name:-${__db_summary_to_operation__:-unknown}}`	`${db.query.summary}`

Helper functions

Helper functions are references to bits of attribute mapping logic which are more involved than simple attribute references.

Function	Description
`__http_error_status_code_or_null__`	Returns string value of `http.status_code if >= 500`
`__rpc_error_status_code_or_null__`	Returns string value of rpc.grpc.status_code if in set: `[2,4,12,13,14,15]`
`__db_summary_to_operation__`	Returns first word in `db.query.summary` in set (case insensitive): `[alter,call,create,delete,drop,exec,execute,insert,merge,select,set,update]`
`__db_summary_to_sql_table__`	Returns first word in `db.query.summary` NOT in set (case insensitive): `[alter,call,create,delete,drop,exec,execute,insert,merge,select,set,update]`
`__null__`	Placeholder for `null`

Golden signals

The golden signals of throughput, response time, and error rate appear in several places throughout the OpenTelemetry APM UI. When used, they are computed as follows:

For metrics, the queries assume data conforms to the HTTP metric or RPC metric semantic conventions.

For spans, the queries are generic, utilizing only the top level span data model. Spans are counted towards throughput and response time if they are root entry spans into a service, computed using a heuristic of WHERE span.kind = server OR span.kind = consumer. Spans are errors if they have a status code of ERROR (for example, otel.status_code = ERROR).

Narrow data with filters

Several pages include a filter bar, with options like Narrow data to.... This allows you to filter queries on the page to match the criteria. For example, you might narrow to a particular canary deployment by filtering for service.version='1.2.3-canary'. Filters are preserved when navigating between pages.

Golden metrics

Golden metrics are low-cardinality versions of golden signals data, such as HTTP/RPC metrics. They populate various platform experiences, including the Entity Explorer, Workloads Activity page, and Change Tracking Details page. These metrics use names like newrelic.goldenmetrics.ext.service.*.

중요

Historically, the OpenTelemetry golden metrics were calculated from spans. Spans are usually sampled, so they only provide a partial picture. Now that metrics are broadly available, golden metrics are calculated using metrics data rather than span data.

Prerequisites.css-21sua1{background:none;border:none;width:0;padding:0;}

Find your OpenTelemetry services

팁

How OpenTelemetry works with New Relic APM

Data mapping process

Benefits for existing New Relic users

중요

Data sources and prioritization

How transactions are derived from OpenTelemetry data

중요

Transaction names

Unknown transaction name

Navigating the APM experience

Summary

Customizing Apdex Target

Distributed tracing

팁

Service map

Transactions

Transaction traces

Segment breakdown

Databases

Time consumption by caller

중요

External services

Time consumption by caller

JVM runtime

Go runtime

Logs

Errors inbox

Span View (Legacy)

팁

New Relic APM metrics derived from span data

Segment breakdown

Database calls

Messaging systems

OpenTelemetry Ruby

APM Service Metrics

Metric resource attributes

Metric: apm.service.transaction.duration

Metric sources

Metric: apm.service.transaction.overview

Span sources

Metric: apm.service.external.host.duration

Metric sources

Metric: apm.service.datastore.operation.duration

Metric sources

Helper functions

Golden signals

Narrow data with filters

Golden metrics

중요

Prerequisites

Metric: `apm.service.transaction.duration`

Metric: `apm.service.transaction.overview`

Metric: `apm.service.external.host.duration`

Metric: `apm.service.datastore.operation.duration`