Investigation conclusion: January 31, 2024
This is our final update to this security bulletin describing our November 2023 security incident involving unauthorized access to our staging environment. New Relic, leading cyber experts, and forensic firms conducted an extensive investigation into the incident, and together have provided updates to this security bulletin as information became available. The investigation has concluded and we are sharing additional information about our findings.
Background
In November 2023, New Relic became aware of unauthorized access to our staging environment—an internal environment that provides visibility into, and information relating to, our customers’ use and operation of our services for troubleshooting purposes (“Staging Environment”). The Staging Environment maintains New Relic’s own observability data, including logs, events, traces and other diagnostic files, ensuring that we have visibility in the event of a failure in our customer facing Production environment. Notably, telemetry and application data sent to New Relic by our customers in their use of the New Relic platform does not reside in our Staging Environment.
Upon learning of the unauthorized access to the Staging Environment, we took immediate action to assess the integrity of internal applications, systems, and infrastructure. We activated our incident response plan and engaged several third-party cybersecurity experts to conduct a thorough investigation into the impact to our customers and our business.
During the course of the investigation, we learned the unauthorized actor used stolen credentials in connection with a single New Relic employee account to gain access to the Staging Environment. The unauthorized access occurred shortly before the completion of a planned migration of the remainder of our employees to our enhanced User management model for added security.
New Relic immediately revoked access to the compromised employee account. We also analyzed the potential impact to customers by searching the Staging Environment for passwords, API keys, user identifiers, including usernames and other customer data. Our investigation also confirmed there was no lateral movement from the Staging Environment to any customer accounts in separate environments or to New Relic’s Production environment.
Additional steps to harden environment
We took additional steps to remediate the impact of the incident, including by redacting secrets out of our logging rules, and took steps to further harden our systems, such as implementing additional layers of technical controls, enhancing our network access controls, and accelerating the migration of our remaining employee users to our enhanced User management model. These additional steps were taken and completed.
Incident findings
Our investigation into the Staging Environment incident is complete and has revealed the following:
- The unauthorized actor utilized a single New Relic employee account to gain access to New Relic’s Staging Environment.
- All activity by the unauthorized actor within New Relic’s Staging Environment has been comprehensively identified and reviewed by New Relic and our industry-leading forensic firms.
- Between October 24 and November 15, 2023, the unauthorized actor executed specific search queries and exfiltrated these query results from the Staging Environment.
- The last observed unauthorized activity in the Staging Environment was on November 16, 2023. There is no indication of persistent access by the unauthorized actor in New Relic’s Staging Environment.
- A very small percentage of our customers were impacted by the search queries executed by the unauthorized actor.
- There is no indication of lateral movement from our Staging Environment to any customers’ New Relic accounts in the separate production environment or to New Relic’s production infrastructure.
Tactics, techniques, and procedures (TTPs)
In support of our industry and community as a whole, we are sharing the tactics, techniques, and procedures (TTPs) utilized by this unauthorized actor so that our community can leverage this information to identify potential risk to their environments. These TTPs were discovered and confirmed through the investigation performed by New Relic in partnership with forensic and cybersecurity experts.
The unauthorized actor used the following tactics, techniques, and procedures (TTPs):
- Credential stuffing;
- Use of Protonmail (proton.me) for communication;
- Use of VPN services including NordVPN for access to public services; and
- Programmatic Data Extraction using APIs.
Eradication and remediation efforts
We understand that the best defense starts right here at New Relic. New Relic took a number of actions to eradicate the unauthorized actor’s access during the incident, including:
- Revoking access to the compromised employee account immediately;
- Blocking indicators of compromise associated with the attack;
- Further hardening access controls and credential theft defenses, leveraging an industry-leading security toolset; increasing our capacity to monitor security across our entire enterprise; and
- Providing additional cyber education and awareness to our employees.
Customer recommendations
We have completed our proactive outreach to customers whose accounts were impacted from this incident in order to help them understand the impact and to communicate suggested remediation steps. If you have not received specific instructions regarding your systems, there is no action you need to take. Customers should review our Security bulletins and Security guides for best practices. There are no additional measures you need to take beyond what has already been communicated with you.
We regret any inconvenience this incident caused for our customers. Our CEO, CTO, and CISO are aligned on the future state of security at New Relic. They share the same commitment to making broad improvements to our security posture, and specifically to preventing the same type of incident from occurring in the future. We have talked to many customers, including those not impacted directly by this incident, and have shared both our commitment to do better and the significant enhancements we have made to our security posture. We will continue making long term investments to earn back the trust of our customers. We deeply appreciate the understanding and support that customers have shown.To all of our customers—we look forward to our continued work together.