Pen Testers Cannot Scale To Enterprise Need!

As reported in Information Week, several Black Hat researchers have noted that humans must qualify application vulnerability scanning results. It’s hard to find this statement as “news” since I’ve been saying this publicly, at conferences for at least 4 years. I’m not the only one. Ask the vendors’ Sales Engineers. They will tell you (they have certainly told me!) that one has to qualify testing results.

“At the end of the day, tools don’t find vulnerabilities. People do,” says Nathan Hamiel, one of the researchers.¬†

Yes, Nathan, this has been the state of affairs for years.

The status quo is great for penetration testers. But it’s terrible for larger organizations. Why? Pen testers are expensive and slow.

I agree with the truisms stated by the researchers. If there were 10’s of thousands of competent penetration testing¬† practitioners, the fact the the tools generally do NOT find vulnerabilities without human qualification would not matter.

But there are only 1000s, perhaps even less than 3000 penetration testers who can perform the required vulnerability qualification. These few must be apportioned across 10s of millions of lines of vulernable code, much of which is exposed to the hostile Public Internet.

Plus which, human qualified vulnerability testing is slow. A good tester can test maybe 50 applications each year. Against that throughput, organizations who have a strong web presence may deploy 50 applications or upgrades each quarter year. Many of the human qualified scans become stale within a quarter or two.

Hence this situation is untenable for enterprises who have an imperative to remove whatever vulnerabilities may be found. Large organizations may have 1000s of applications deployed. At $4-500/hour, which code do you scan? Even the least critical application might hand a malefactor a compromise.

This is a fundamental mis-match.

The pen testers want to find everything that they can. All fine and good.

But enterprises must reduce their attack surface. I would argue that any reduction is valuable. We don’t have to find every vulnerability in every application. Without adequate testing, there are 100% of the vulnerabilities exposed. Reducing these by even 20% overall is a big win.

Further, it is not cost effective to throw pen testers at this problem. It has to either be completely automated or the work has to be done by developers or QA folk who do not have the knowledge to qualify most results.

Indeed, the web developer him/herself must be able to at least test and remove some of the attack surface while developing. Due to the nature of custom code testing, I’m afraid that the automation is not yet here. That puts the burden on people who have little interest in security for security’s sake.

I’ve spoken to almost every vulnerability testing vendor. Some understand this problem. Some have focused on the expert tester exclusively. But, as near as I can tell, the state of the art is at best hindered by noisy results that typically require human qualification. At worst, some of these tools are simply unsuitable for the volume of code that is vulnerable.

As long as so much noise remains in the results, we, the security industry, need to think differently about this problem.

I’m speaking at SANS What Works in Security Architecture August 29-30, Washington DC, USA. This is a gathering of Security Architecture practioners from around the world. If your role includes Architecture, let me urge you to join us as we show each other what’s worked and as we discuss are mutual problem areas.



3 thoughts on “Pen Testers Cannot Scale To Enterprise Need!

  1. Strategy consultants that have penetration-testing experience will scale to Enterprise need when decision-makers realize their value and promote them in the proper roles necessary to complete an application security program (including application security risk management).

    Without penetration-testing experience, application security risk management is an exercise in futility.

    Does this mean that the strategy consultant’s entire job will be performing application penetration-testing or secure code reviews? Absolutely not, it would be a Six Sigma system where in the first few years, many of these kinds of assessments (especially source-code assisted penetration-tests) would be done, but less increasing as agility and information optimization occur in the software lifecycle.

    For example, your average skilled app pen-test and secure code review expert would perform 40 or more assessments for an organization in the first year, 30 in the second, 20 in the third, and 10 on-going from there — taking up about an equivalent percentage of their time and effort.

  2. Andre,

    Great comment. Thanks! I defintely agree that application pen test is one portion of an ongoing and far more holistic application security programme.

    I think your decreasing scan rate is based upon the assumption that secure coding knowledge grows as the programme matures?


  3. Brook,

    Only when the organization can successfully “climb the wall” —

    Otherwise, there is no maturity and the organization will literally increase the costs of their application security program over time.

    Also — where did you get the word “scan” from? Every modern app pen-tester has secure code review skills (and vice versa). Reliance on scanners is fleeting. I consider “scanning” to be about 3 percent of what I do, at best. The industry prefers the terminology “automated web application security testing” for internal use such as Cigital ESP — or perhaps SaaS-based services such as WebInspect On-Demand (through HP Software’s Fortify division) or Veracode when speaking about outsourced solutions. Application security as a service, just so you know, doesn’t involve running scans (although certainly people will continue to think it does and perhaps reference it by that name).

Leave a Reply