What to Look for When Evaluating an Application Testing Provider

Capabilities and expertise

1. What types of tests do you offer? 

The answer should be: any type of test you need, on-demand, at scale. Available testing options should span from automated to in-depth manual testing. Industry analysts will tell you no single testing tool can detect all application security vulnerabilities. Vendors should have the ability to utilize multiple best-of-breed tools, with customizations that match your business needs.

Keep in mind: automated testing alone is not sufficient to provide a complete picture of your vulnerabilities. To defend against multi-step attacks or ones that involve social engineering, your vendor should be able to conduct in-depth manual testing to mirror the perspective of a hacker.

2. How will your tests match my risk profile?

Your vendor should have the expertise to apply different testing strategies based on the risk level and unique requirements of each of your applications.

The right vendor will help you create a full inventory of your applications and rank them according to security risk. They’ll design a testing plan so you can focus time and money on the things that matter most.

3. When you find vulnerabilities, how will you help me fix them?

Classic application security testing vendors consider their job to be just that—running tests. Vendors with a holistic approach provide remediation guidance to empower you to fix issues and address causes so fewer security issues ever reach the testing phase.

Make sure you understand how reports are created and verified. You’ll be more confident if you know every testing report is reviewed by a security expert to eliminate false positives. The top vendors will also include contextual remediation guidance in all reports along with the vulnerabilities they find.

Vendors should review findings with you directly. They should include developers in report read-outs to detail causes of vulnerabilities and remediation advice. Even after the initial test read-out, they should provide on-demand live remediation support.

4. How well do you know the security compliance requirements for my industry?

If your applications are subject to industry-specific requirements (PCI DSS, HIPAA, etc.), make sure the vendor includes compliance testing. As regulations are becoming stricter and penalties for non-compliance are increasing, it’s essential that your vendor is proactive in providing guidance to you on any actions you need to make.

Your vendor should help you do more than simply meet minimum requirements for compliance, by including compliance as part of a broader application security strategy.

5. How will you demonstrate success?

To see whether application security testing has been worth the investment, consider how your vendor’s approach will help you answer the following:

• How quickly are tests run compared with any previous system you used?
• Are testing methods identifying vulnerabilities previously missed?
• Are you saving time provisioning secure production applications?
• Are your developers improving over time? What is the defect density of new code?
• Is design improving? Or are tests finding the same flaws over and over?
• Do developers now have time they can use on other projects?

Flexibility and ease of use

6. How easy is it to run tests?

Once you decide to invest in application security testing, you’ll want to get going with minimal fuss so that you can start seeing results right away.

Make sure your vendor provides sufficient resources to jumpstart your testing program. Make sure your vendor insulates you from the complexities of running an optimized application test.

Dig into the details:

• What does it take to request a test on one or more of your applications?
• Do you need to schedule in advance and confirm testing resources are available? Or, does the vendor allow you to schedule tests and set testing windows?
• Can you specify which applications you want to test and the type of tests you want for each?
• Does only one person have the ability to request tests, or can anyone on your team request a test when needed? 

7. How will I know what kinds of tests have been run on my applications?

It is important that you have control to determine when tests take place and what type of tests your vendor runs on your applications. Make sure you have a transparent way to track your vendor’s performance and an easy way to retrieve tests results and share remediation recommendations. That way, when it’s time to make decisions about budget and answer your boss’s questions about how money was spent, you can clearly demonstrate the work that was done.

8. Do you have capacity to test all of my applications?

Check that your vendor has a coverage model that lets you test your full portfolio at the depth that maps to your application risk profile. Any application in your portfolio can provide a hacker access to reach your most valuable and sensitive data. Even if you don’t test every application at the same depth, it’s important to have a full inventory of your applications and a consistent testing schedule so nothing is missed.