Overcome AST noise to find and fix software vulnerabilities

Unclear security test results create challenges for DevSecOps

One of the most striking findings in the “Global State of DevSecOps” report is the sheer number of security testing tools organizations are using. The survey uncovered that an astonishing 82% of organizations use between six and 20 AST tools. While this may provide comprehensive coverage, a large collection of tools introduces significant complexity when integrating into developer workflows and CI pipelines. Tool proliferation also complicates the interpretation of results and management of your testing portfolio. Ultimately, since a large number of tools introduces a large number of potential points of failure, it can generate inaccurate, contradictory, or redundant results; present conflicting priorities; and inundate security and development teams with unnecessary distractions. This is why the problem is called “noise.”

The survey findings show that even those organizations that are performing security testing efficiently are not seeing efficient issue resolution. Even if the 52% of organizations that report using more than 10 security testing tools are managing those tools with a single interface, they’re still faced with a huge number of results to triage. ​Plus only 9% of survey respondents are getting anywhere near full test coverage (defined as testing 81% to 100% of projects in the test queue). There are lots of tests and lots of results—but 30% of survey respondents report that most of their results are noise.

What causes inaccurate software security tests?

Most AST tools are limited by how well they are configured for the projects and technologies they assess, the structure of their governing policies, and whether they are placed at the right position in the development pipeline. Misalignment on any of these factors will contribute to noisy results.

Although establishing AppSec test coverage across your full project portfolio is important, the greater determinant of how actionable those tests are is your ability to extract clear, actionable insight from your AppSec testing. What the report tells us is that too many organizations are still struggling to sort meaningful information from the mass of testing results. 

How pervasive is this problem of noisy application security tests? A total of 60% of respondents in the “Global State of DevSecOps” report survey said that between 21% and 60% of their security test results are noise. When AppSec testing consistently returns too many alerts, organizations can direct too much of their security efforts toward noncritical issues while genuine threats can be overlooked. 

It's important to remember that there are role-based differences that affect the perception of what is, and is not, security noise. It is no surprise that security personnel perceive a high percentage of noise in security test results since they sit toward the top of the review funnel. And because they then present development teams with prioritized results, it makes sense that development and engineering teams perceive a lower level of testing noise. Seventeen percent of developers and engineers, however, report that they simply don’t have enough visibility to determine what is noise and what is not. This means it’s quite possible that they may be getting bogged down by unclear, potentially irrelevant issue data without even knowing it.

Are application security testing tools right for DevOps?

To directly address this question, it’s important to understand that not all security tests, nor the tools that perform them, are inherently an impediment to DevOps. Problematic noise arises when tools are improperly configured, inefficiently integrated into key stages of the SDLC or CI/CD pipelines, or when they’re not developer-friendly. 

Tools can be improperly configured in the sense that they are not working as intended, or they can be improperly configured in the sense that they are not the most appropriate security mechanism for that specific project, team, or organization. When either happens, you are likely to see a drastic reduction in the efficiency of application security tests and a sharp increase in the resentment development teams exhibit toward them. To achieve true DevSecOps, AppSec needs to impede neither DevOps nor security. “Noise” is the resulting distraction and delay from slow or irrelevant scans. 

Another common source of security noise in DevOps is inefficient tool integration. Even as DevSecOps becomes commonplace, we still see teams relying on late-stage testing. Late-stage testing is inefficient because by the time vulnerabilities are detected, development engineers have moved on to new projects or branches. Asking them to return to work they’ve already completed interrupts everyone’s workflows. In addition, since tests often reveal different insights when performed at different stages, the best time to derive those insights is when they are introduced into the project, and the best time to make code fixes is before the issues propagate downstream.

Automation is the future of DevSecOps

While the “Global State of DevSecOps” report tells us that DevSecOps is becoming more standard across all industry sectors, the idea that security testing impedes development persists, but that perception varies depending on one’s role in the organization. 

The report shows that 65% of AppSec team members surveyed felt that pipelines are marginally or severely impeded by tests. The results do not distinguish whether this is due to the AppSec professional’s proximity to the testing process or the pressures placed on them to expedite the review process, but surprisingly, 58% of development team members agree with them. In addition, 68% of C-suite representatives surveyed also agree that testing impedes development on a moderate-to-severe level. 

When you drill deeper into the report results, however, 57% of the cohort who report development delays due to testing manually prioritize issues, while 41% use testing automation. The report shows that among those who feel that AppSec does not slow down the release pipeline, only 23% prioritize issues manually, while 69% automate their testing. The takeaway: Most of those who are not impeded by AST automate their tests, while most of those who are impeded rely on manual activities.

Moving forward, those organizations that will be most successful at implementing DevSecOps will likely be those that can effectively automate their AST tool stacks and integrate automated testing throughout their development pipelines.

How do you reduce noise in your DevSecOps program?

Data gathered by the “Global State of DevSecOps” report indicates that there are four primary ways to eliminate noise within security tests and evolve your DevSecOps program so that you’re developing securely at speed. 

• Meet developers where they are. By building testing and insights directly into development tools and workflows, you make security frictionless and minimize developers’ desire to circumvent security.
• Integrate and automate testing across the stack. By integrating automated testing aligned to risk tolerance policies, you establish security gates, tighten feedback loops, and eliminate implicit trust in AI.
• Cultivate developers’ security capabilities. By empowering developers with risk awareness and AI-powered fix guidance, you can cultivate higher development standards with every release and fix vulnerabilities at speed.
• Plan for evolution with a strategy that lasts. By unifying policies, insight, and testing atop a flexible application security testing platform, you can build a DevSecOps strategy that will grow with your business.

The report indicates that organizations that see the most success from DevSecOps will likely be those that can effectively streamline their AppSec tech stacks, leverage AI responsibly during development, and reduce noise in security test results to foster closer collaboration between security, development, and operations teams. 

The DevSecOps journey is far from over, and this year’s “Global State of DevSecOps” report helps define that path so you can navigate it more securely without slowing down.