Reporting

The whole point of security testing is to provide a report on the risks for the business that have been identified. Business want to be assured that the changes or the state of the system keep the resulting risk within risk appetite.

Reports adapted to the audiences

The actual tests result in technical reports on weaknesses, vulnerabilities and severities. That level is too technical for the business so an interpretation of that in context should produce a business risk impact report. The business security analyst that defined controls and the tests to be done now has to take the results and cast them in the evaluation of the controls that needed testing.

Once upon a time in a country far away my report required to split the outer and the inner security over different devices in separate zones. The responsible asked a simple question: "This is going to cost me $$$ to acquire, change and support afterwards. Are you sure you stick with your requirement to split?". It is a fair question and stays relevant today. It brings the real world into the equation.

Another client CISO truly understood why he was paid. The report identified "a really bad vulnerability" in the companies webserver and his reaction was: "So what? Let's discuss the real problems first." The public website was unconnected to their core business web servers and managed by an agency. Sites do get hacked, errors do happen, no clients or providers or … were impacted. Just fix it, no shame or hiding or making a fuss about it: communicate openly.

Reports adapted to the context

This point is not a simple one. Typical technical reports will include a vulnerability rating according to some methodology, like the CVSS. The advantage of CVSS is that its scores will be rather consistent between similar technical risks.

The drawback of relying solely on CVSS and similar ratings is that they lack specific context.

Consider reputational risk. To maintain your reputation as a top security aware company, even minor mistakes on the public website are a no go. There are websites that allow you to check on the security posture of domains. As a top security company you want a full green result from these checks.

The reverse also happens: a seemingly serious bug is not exploitable in the given context. One frequent cause is that the website is behind a WAF (web application firewall), yet, often testing is done on the webserver directly.

The costs of fixing my also be used to increase the priority of the fix: if it takes less time to fix the issue than to explain it, just fix it. An example of this are TLS configurations that need a cleanup. Exploits are unlikely, a higher priority is following the principle of better safe than sorry.

At some point the technical report content has to be lifted to the business level. The business owner does not need technical vulnerabilities scorings to decide on budgeting and priorities, what matters is to understand what can go wrong with his application and how likely it is.

Continuous improvement: reporting test metrics

Measure what you want to manage. Some objectives of metrics related to security testing are:

• measure comprehensiveness of security testing
• follow progress in the state of security
• quantify success in training and awareness campaigns
• track elimination of discovered vulnerabilities
• monitor security incident response capabilities.

Basic metrics

Remediation assistance

Part of the reason why vulnerabilities show up is that the implementer or configurator did not know or understand why it is a mistake. To ensure that the issue will be fixed, and the error will not be repeated, the responsible will have to be briefed on the issue and the possible remediations. This is another form of reporting.

A frequent question is: how can the identified problem by fixed?

The wrong expectation is that in the end, the tester is fixing the problem. The responsible for the problem should invest in understanding why it is a problem, and troubleshoot how the issue got incorporated, and how to avoid it. That knowledge should be distributed in the teams for improvement.

From OWASP there is very good guidance on many application security issues and their fixes. That can provide a baseline for remediation.

It may be wise to organize awareness training on these issues by specialized trainers for both didactical and practical reasons: security testers should not spend too much of their time providing awareness training.