In December of 2021, software companies around the world discovered that Log4j, an open-source logging library bundled in many software packages, contained significant vulnerabilities, one of which was a 10/10 on the CVSS scale. The Federal Trade Commission called the library “ubiquitous.”

Jon discovered that the offending module recurred throughout their build. “The more we dug,” Jon explained, “the more we realized this bug was everywhere and nowhere at the same time.”

Nutanix moved quickly, despite having multiple build and artifact management systems, as well as a large monorepo with many component branches and hundreds of git repositories. In under four days, Nutanix was able to deliver patches to its customers that fully remediated the Log4j vulnerability.

Speed was of the essence, but the timing of the Log4j news, which broke right before many employees go on vacation for the winter holidays, didn't make things easy.

However, Nutanix armed its engineers with Sourcegraph. Within a couple of days, a few Sourcegraph queries identified every instance of the Log4j vulnerability.

With the Log4j 1.x vulnerability, for instance, codebases were only insecure if they used JMSAppender. Jon used Sourcegraph to see where JMSAppender existed, fixed it, and sent out a release. “That took almost less than five minutes,” Jon said. Sourcegraph released a blog post that explained how other companies addressing Log4j could use code search for similar benefits.

This speed gave the team a head start on mitigation.

Deploying these fixes required quality assurance and testing as well as discovery and fixing.

With the help of Sourcegraph, Nutanix was able to release three back-to-back patches relatively quickly compared to other companies. Nutanix's customers reported satisfaction with both the speed of the patches and their quality.

“We tried to take a measured approach,” Jon said. “We wanted to get things done quickly but without completely flying by the seats of our pants.”

Tracking down the Log4j vulnerability was, in Jon's words, like “herding cats who were herding mice at the same time.”

Without Sourcegraph, Jon would have either been tracking down whoever built each component to ask them how and where they used Log4j or stumbling through all of the company's repositories.

Using Sourcegraph, Jon discovered every instance of Log4j and was fully confident in the results. “It's nice,” Jon said, “when you can just run a report and say, 'Here it is,'' or 'Here it isn't.'' It's much better than having to say, 'Well, boss, I think we got it all.’”

Nutanix needed that confidence because of its sprawling codebase. One thing that made Log4j especially complicated for Nutanix—as it does for other large-scale enterprises—is that there were multiple source control systems in play. Sourcegraph provided them with “unified visibility,” according to Jon. “I can't imagine the pain of having to do that either with grep or OpenGrok,” he added.

Confidence spread from Jon to the rest of the team and throughout the company. With search contexts, Jon was able to share relevant contexts and queries, showing the team how they could verify whether a given Log4j instance was present or absent. He could show them precisely what they changed.

“We used Sourcegraph contexts to see specifically where a service was at any given point in time,” Jon said. Without Sourcegraph, the team would've had to use code scanning, which takes a lot of time, or manual build inspections, which aren't foolproof.

With Sourcegraph's help, Nutanix was able to transform a trust-threatening risk into a trust-building opportunity. Their customers, Jon explained, were worried about hundreds of other vendors, all of whom were likely affected by Log4j.

“That's hopefully something customers will remember us for,” Jon said. “We quickly either provided them with clarity or gave them a line on the next available patch because we were able to identify the issue and start fixing it ASAP.”

Jon explained that, at many enterprises, dependencies are unseen and forgotten. “You might check in on a dependency and find it hasn't been reviewed for four, five, six, or even ten years. But it works, so why update it?”

Log4j is one of many reasons why monitoring and updating is now a renewed priority. “Companies will have to be more diligent,” Jon said. With Sourcegraph, Nutanix is prepared to find and fix the next vulnerability.