Introduction: business viewpoint

Business has the responsibility to use secure IT solutions. Whether solutions are bought or created their security must have risks within the risk appetite of the company.

Business confidence should be based on proof. Security testing of a system is delivering evidence of the security posture. Security testing includes very diverse methods of testing, in one of the many IT environments, covering specific objectives.

This document presents available security test choices to facilitate discussing a test plan with the security function. The document intends to make the link from objectives to specific tests clear.

To track all testing, a heatmap of all IT components, in the scope of a business owner, and their relationships, are mapped on test data. These include tests performed and planned, date of testing, and test outcomes. The result is a comprehensive overview to manage security testing.

Introduction: security viewpoint

Scope

The CISO function puts a lot of effort into implementing preventive and detective controls for the organization. For judging the security posture of the infrastructure and applications, technical testing provides a view of the actual situation. We will use the shorthand "security testing" for the rest of this document.

There are other forms of testing that will not be addressed here, like red team attacks, social engineering attacks, and physical intrusions. Regulations for critical infrastructure do require these as well.

Overview of the topics for security testing

Why?

There are many triggers for security test, each with their specific targets and objectives. The test scope and the test coverage may not be clear to all parties. That uncertainty could produce a result that does not meet the expectations of the requestor, and worse, that discrepancy may remain hidden to both tester and requestor.

What?

There is a long list of security tests that you could and should perform. Some are more common than others. The scope of the test can be expressed in targets: which targets are included in the tests? The coverage defines precisely the tests that must be done. Finally, the tests may be sampling or comprehensive. In the former case a representative set of items is tested, whereas in the latter, all items on the checklist for all relevant target components are tested. A good example of comprehensive testing is a configuration review against a standard which can be largely automated.

How?

In almost all cases the test duration is fixed (time-boxed testing) which implicates not everything may be tested comprehensively. It may be restricted to sampling the targets as well as the tests on the chosen targets.

Where?

The choice between production, pre-production, testing, or development environments for testing matters for the usefulness of the results. The build-security-in approach embeds testing in all environments.

Reporting and metrics

The reporting on the findings is de technical basis for higher level reporting up to the accountable stakeholder and audit. An action plan to remedy all findings with a risk exceeding the organizations appetite is another deliverable with input from the testing report.

Metrics on the findings allow management to measure progress.

The typical phases and steps of security testing

The phases of a generic security test are:

- Trigger

There is a reason why a test is organized.

- Objective and context

What assurance is sought?

- Coverage

There is a lot of variation in coverage possible, therefor the coverage must be precisely defined.

- Tester selection

Security tests are getting more and more specialized. Pick the right skills for the job.

- Planning

Time matters; it is measured in days and hours. Lots of conditions must be met at the start.

- Execution

Security test try to break stuff. Collateral damage is possible. Be prepared.

- Reporting

Meaningful reports for all stakeholders are key.

- Remediation

Do not just point at the problem, guide to the solution.

- Continuous improvement

Spending money on recurring issues is sad. It brings no joy to a pentester either when find the next SQL injection vulnerability is uncovered. My moto: A waste of my time, and a waste of your money.

In addendum you find a possible RACI matrix for the typical stakeholders.

Test triggers

Security testing triggers

Functional security testing like authorization enforcement, logging of security events etc. the testing should be part of standard Quality Assurance. Secure configuration standards compliance should also be business as usual.

In other cases the motivation for a security test is grounded in the security policy, that itself may be driven by regulations. Some typical triggers can be for instance:

- Changes

The introduction of new systems or new versions of systems may introduce new issues or make others more severe. New issues may arise in conjunction with existing systems, the configuration of security may pose problems due to unfamiliarity with the impact of settings.

The modifications may also solve previously reported vulnerabilities, which typically can only be closed after a new test.

- Regulatory requirements

Many industries are (heavily) regulated, including more security controls over time. Their license to operate may depend on acceptable test results. Providers of systems to any regulated organization likely requires the provider to have attestations from testing the system.

- Policy of the organization

The organizations' policy typically contains controls related to security testing, to prove to their people, their clients etc. that they take care of their security.

- Incidents

Incidents uncovering vulnerabilities not found or not covered by previous testing should lead to a test on other systems that may have the same issues.

- Security intelligence reports

Intelligence-based prioritization of new or extra tests to deal with active or upcoming attacks may be required, and this focus on real and present danger just makes sense.

When and how to define testing objectives?

The principle:

For every definition of a control, it must be defined simultaneously how compliance will be established, in sufficient, practical ways.

The way to look at it is putting the change or creation of a component between security brackets:

• The opening bracket is where you define security controls, and how compliance shall be validated
• The closing bracket is where you check compliance with the controls according to the rules set forth initially, with the security tests defined in the opening bracket

If you open a bracket, it must be closed.

No surprise attacks

This way of working avoids the "surprise!" effect that is typical: security testing finds flaws or oversights, yet, the project was unaware of the issues till they were reported.

It can be underestimated what it means to implement a control. Controls may be defined without realizing the implications or how to do it at all. A single control may touch upon many targets and different type of targets. If that is not made explicit, it is also hard to know how to check compliance with the control.

Checking compliance through security tests implies defining which tests on which targets are providing reliable compliance assurance. If there is no way to establish compliance, the control

Specific testing objectives

THE security test does not exist

There may be some misunderstanding around security testing. It happens often that a test request sounds like this "Please do a security test of XYZ, before the end of next month.". This request misses the point that many choices exist for what to test, where to test, and who does the testing, and is maybe optimistic on the when to test, and may forego the possibility the test could be failed.

The clarification "pentest" does not help. There are still too many variations and assurances that could result. It has no precise meaning so a mismatch between test and expectations is likely.

Identify the objectives associated to testable targets

The main parameters to define before testing are the security objectives, the scope specifying the targets, the controls to be tested, the test coverage, the expected time box, and the restrictions on testing. It is up to the organization to define these parameters for the planned pentest, and then match this plan with specific tests and assign these to qualified testers.

The objectives are mostly defined by two parties: the business owners of the targets and the security function. Business owners are driven by compliance and commercial reasons. They know the regulatory environment of the organization and the security requirements in these regulations. They need evidence that their systems meet those requirements as a minimum. Security requirements are defined in approved policies, standards, guidelines and principles.

Check if planned tests would meet the objective

Each security requirement relevant for the targets should be tested to have sufficient assurance for the desired security of the targets. Testing requirements can be factored into specific tests meeting typical testing competences.

See addendum: common tests

Test coverage

Once the targets, the requirements and the testing types are defined, test coverage is next. The following is a selection of bullet points showing the diversity of options, without details that would deviate from the main objective of this document.

Testing general security functionality:

The list would include:

• Security Logging and safekeeping
• Audit logging
• Monitoring capabilities
• Management access control
• Security configuration
• Key management
• Authentication
• Access control

A generic security test suite might include security test cases to validate both positive and negative requirements for security controls such as:

• Identity, authentication & access control
• Input validation & encoding
• Encryption
• User and session management
• Error and exception handling

Targets

Different types of targets require different testing.

Network:

• Network devices: configuration guidelines and policies
• Communication protocols: security configurations
• Filtering: rules
• Segmentation/zoning/zero trust principles and practice
• Event generation
• DLP

Systems:

• Services running
• Hardening guidelines and compliance
• Software control of versions, licenses
• Configuration guidelines and verification
• Malware protection
• Access control, esp. Privileged access control
• Security Event generation
• Patching policy and practice

Applications:

• Installation rights
• Secure installation guidelines and configuration
• Patching, version management
• IAM issues
• Connection security
• Code issues

Choose the right testers

Independent testing

The first question to answer is how independent the testing needs to be from the business owners of the targets. For regulatory testing the testers' organization must be independent from the organization owning the targets, so at the highest level. It also requires that the target creation, maintenance or operation is not even partially done by the testers' organization.

At the other end, in agile mode, the tester may be part of the team. He is responsible for security testing and his results may be inspected like for any other job to see if it is done professionally.

Security testing may be a part of the security organization within the organization itself. The level of independence depends on the organization. If security is part of IT, there may be conflicts of interest.

Testers' specializations

While testers may have broad competences, they usually focus on specific areas: DB security, system hardening, switches and firewalls, webservers and reverse proxies and web application firewalls, email systems, cloud connectivity, mobile clients, SAP systems, data lakes, mainframe, … There is a long list of testing disciplines and expertise.

Please also check specific requirements for security tests that are mandatory for your industry branch. These may include testing providers to be accredited for the required security testing.

There are some clusters of tests that have synergies. Check your testers for the specific skills, they may have narrower or broader skill sets.

Infrastructure testing

The test based on running scans on the targeted networks is a core one. The findings can be a annotated huge list, ordered by severity. The severity ratings are often context-agnostic: they would be the same regardless of the system and its location in the infrastructure.

There are many offerings for these scans, with a wide variety of impact to deploy, depth of view and contextualizing options.

Outdated versions and critical vulnerabilities must be investigated and be put in context. These adjustments are the basis of a prioritized to do list. While scans easily list thousands of issues, the priority list tends to be much smaller.

The most important skills of the scan-based testing is being able to explain issues and severities and decide, in cooperation with the infrastructure team, the right priorities and plans to resolve those.

Application testing

Web applications

Today the applications that are most likely to be tested are web applications. They are externally facing so security issues are very visible, they are providing important business functions for the organizations, they are used for supporting remote access by providers, customers and staff.

Contrary to infrastructure testing, the non-profit organization OWASP was founded early in the rise of web applications and has since been a key resource for all application testing. The most famous series of documents are the OWASP top 10 (web) application security issues. Please note that limiting your testing to the OWASP top 10 issues is the very basic, essential level of testing only.

They also have much more elaborate guidance on application security and application security testing: the "OWASP SAMM", the software assurance guide, and the "OWASP Application Security Verification Standard", and the "OWASP web security testing guide".

Office automation/end-user computing/shadow software

Office tools like MS excel and MS access can be used to build applications. They have their specific risks and they require specific skills to do security testing. These applications are initially build with minimal code, and using basic skills, yet they can grow to serious levels of complexity and large impact if broken.

Specific classes: mobile apps

Mobile apps are yet another special category within applications. OWASP provides testing support.

"The OWASP Mobile Application Security (MAS) flagship project provides a security standard for mobile apps (OWASP MASVS) and a comprehensive testing guide (OWASP MASTG) that covers the processes, techniques, and tools used during a mobile app security test, as well as an exhaustive set of test cases that enables testers to deliver consistent and complete results."

See:

https://owasp.org/www-project-samm/

https://owasp.org/www-project-application-security-verification-standard/

https://owasp.org/www-project-web-security-testing-guide/

https://owasp.org/www-project-mobile-app-security/

Specific enterprise-level software

Many organizations have enterprise level solutions in place. There are certainly databases and possibly data lakes or data mining solutions in place these days. There are widespread systems like SAP systems, there may be mainframes in place, …

These pieces of software come with dedicated solutions and configurations for security. While a generalist may be able to do some security testing, it is best to look for more in depth understanding.

Large in-house systems build by or for the organization have the additional problem that knowledge may be found only within the company, putting the independence of the reviewer/tester in question.

Building management systems and SCADA systems

While many of the systems in this domain are linked to physical security measures, they are actually networked applications with centralized control systems.

They may be sharing infrastructure with the other IT systems, especially network components and network usage. They may be black boxes with little information on the internals. They may not respond well to standard network scans. If they are linked to safety testing must be done by competent people understanding the potential impact of testing scenario's, to avoid any safety risks.

A particular problem is quit common: organizations may share buildings or floors or meeting rooms with other tenants. That typically implies shared HVAC, and also technical infrastructure located in shared locations. Testing any shared infrastructure runs into restrictions and liabilities.

Test planning

Test timeslot

Test timeslot planning depends on the test triggers. All of these triggers come with their own constraints and surprises. That is what makes test planning sometimes a hard job.

The seemingly simplest test is one that is scheduled with a defined frequency. One would expect that you assign slots, do the tests at the defined slot, and you are done. These tests are usually tests in pre-production or production.

There are elements that can get in the way of the original planning. Production test environment may be blocked during certain business periods were any interruption would be very unwelcome. If there is an attack going on the extra alerts and events caused by testing are not welcome.

Pentest related to change projects do have a plan when they expect the system to be in a state that allows correct testing. Experience shows that a plan rarely is stable for the whole lifetime of the project. However, as pentesting is most often executed by an external, independent company, even a shift of a week may be difficulty, especially near the planned date. Ignoring the changed planning means testing an unfinished product and probably dealing with an unstable environment. These elements may render the test results unusable.

For project pentesting there is another frequent problem. The test may fail to approve the result as acceptable. Fixes will be required, as well as a retest. Project planning may not have foreseen the delay. The pentest company may not have resources for the retest at the ideal point in time.

Testing set-up

The what-when-where must be formalized:

• What: a well-defined lists of targets
• When: the timeframe for the testing
• Where:
• Which environment
• On-prem, special facility, remote

And also the how-who:

• How:
• tests to run
• methodology
• Who
• Internal
• External: skills required, (accredited) vendors

The statement of work (SoW) is the formal contract between the tester and the client. It must contain the clear scope of the contracted work related to the:

• What
• How
• Restrictions
• Reporting
• Remediation assistance

Kick-off information is delivered according to the type of test:

• The provided content depends on white box, gray box, black box (multiple possible)
• Target information
• Test credentials, probably multiple (1 per role)
• Authorization model
• Key operations
• Connectivity requirements
• Legal accountability
• Excluded tests (typically: availability attacks are excluded)
• Secure Information exchange channels
• SPoC information is double checked
• Incident reporting procedure

Logistics

Assign a single point of contact (SPoC)

The first action is to establish a (SPoC) for both the organizing team and the testing team. Both need to give priority to any calls/texts/emails from the other SPoC. They might find themselves in a race with incident responders.

Secure information exchange

During most type of tests confidential and even secret information, like credentials and vulnerability reports, may have to be exchanged. If the tester is of the same organization there will be solutions for that. In between organizations it is more difficult to agree: both parties may have solutions they prefer to use.

The environment selected for testing plays a major role in the preparation.

Testing in production and pre-production environments

Pre-production may share almost all the management functions like logging, monitoring, and incident handling. Therefor, they are treated together.

Collateral damage during the test: logging and monitoring

Tests on production impact any logging up to and including the targets, and possibly systems the targets connect to. Monitoring on the targets or connections or infrastructure used to connect for the testing is impacted.

Even if the testing is planned in stealth mode, or with restrictions on authorized disruptions like denial of service, it cannot be excluded that the testing causes incidents on the production. Production incidents start a chain of events and actions with a potential big cost in effort. The organization must be prepared to stop this test-triggered incident handling before it is out of control.

Security weakening: IAM

The authentication and access control components will be those of production. That can get in the way of tool-based testing, if you get to run any tools at all. Introducing backdoors for production testing has its drawbacks as well: backdoors tend to be abused and weaken the security.

If scenarios involve 4-eyes control, it is very tricky to allow the testers to have multiple accounts to test those scenarios on production. There is a good reason to demand 4-eyes control.

"Rogue" connections

To test systems in production requires connectivity to the target systems from sources that would normally not be authorized. The extra connectivity can be created physically, using dedicated cabling to specific physical locations. Establishing connectivity can be logical via enabling limited routes to the targets from locations deemed safe enough, like a room with strictly limited badge-based access.

Testing in test environments

Testing environments are by nature much easier to work in. The key issue is to ensure a stable set-up for the security tests. They are going to test all parts of the targets so the complete test set-up for the targets must be exclusively reserved for these tests. Another reason for this exclusive access is the log inspection to see what was logged to evaluate incident response information, and for quickly seeing impacts of tests on the targets without "clutter" from other tests.

Access to the test environments (accounts, credentials) are rarely a problem.

As security testing requires a set-up that is as close to finished as possible, it will be scheduled near the end of testing just before moving to (pre)production. That timeframe is always hectic and often instable. Combined with the exclusiveness desire, conflicting interests with other tests are likely.

Good communication with the test leads and project management is recommended.

Testing in development environments

On the point of access or tooling, the development environment is the easiest. Stability on the other hand is naturally at risk. Developers will likely not be blocked from access during the test, just informed not to change stuff impacting the test, not a strong safeguard.

Good communication with project management is recommended.

Test execution

During the test execution the SPoCs of the organization and of the pentest team must be able to reach each other quickly. Things can go wrong despite careful preparation.

The kick-off is the checkpoint to so if all arrangements have been made, The SPoCs briefed, the restrictions are clear, all contractual issues have been solved, NDAs are ok, test access is set-up, tested and ready to be used, …

Things that go wrong

The test targets are defined and other systems are out of scope. Unfortunately, it is easy to step outside of the agreed environment.

- The connectivity solution may be shared with other systems and collateral damage may occur

- While deliberate denial-of-service is excluded, security testing walks outside of the usual and may break the targets or systems used to test the targets

- Web pages can contain links to any other page, so while spidering for instance, leaving the targets is easily done

- The underlying database may be corrupted due to the tests, by SQL injection or otherwise

- The authentication and authorization systems might be shared with other set-ups or even production, as managing access in development or test might be considered important enough to use the general IAM solutions

- Tests usually generate lots of logs and may trigger monitoring alert. The logs might get full and that impacts other systems. The monitoring might get overloaded with security events.

- Getting a locked account is business as usual when pentesting. The support team may not like to have to reset the test accounts every 5 minutes.

- Using a pentest laptop on the organizations network may generate traffic to the outside that is blocked by DLP systems, or as other malicious traffic going out.

- (every pentester can extend this list)

If resolving any of these issues takes too long, the time box gets shorter and less confidence in the test outcome may be the result.

High risk reporting

Whenever pentesters find high risk items (according to their evaluation scheme) they must immediately report it. The issue might have been pre-existing the change, meaning it is in the live version. Even if it is not, it is best to continue investigating the impact and the possible fixes as a high priority finding blocks going into production.

Monitoring actions

A pentest can also be used as a test on the detective measures of the organization. As the whole pentest is one big attempt to break in, lots of events should become triggered and logs should be created. Checking if the logs and events are indeed catching what has happened should be included in the pentest review.

Preliminary data

The accountable and responsible people always like to get an preview of what to expect: how many things have been found, and will there be findings delaying their release? The severity ratings may still shift after careful consideration and be different in the report. More important, the adaption to the corporate context can modify the outcome too.

For this reason, the preliminary data may be confined to the CISO team.

Reporting

The whole point of security testing is to provide a report on the risks for the business that have been identified. Business want to be assured that the changes or the state of the system stay within risk appetite.

Reports adapted to the audiences

The actual tests result in technical reports on weaknesses, vulnerabilities and severities. That level is too technical for the business so an interpretation of that in context should produce a business risk impact report. The business security analyst that defined the tests to be done now has to take the results and cast them in the evaluation of the controls that needed testing.

Once upon a time in a country far away my report required to split the outer and the inner security over different devices in separate zones. The responsible asked a simple question: "This is going to cost me $$$ to acquire, change and support afterwards. Are you sure you stick with your requirement to change it?". It is a fair question and stays relevant today. It brings the real world into the equation.

Another client CISO truly understood why he was paid. As a report on "a really bad vulnerability" in the companies webserver he said: "So what? Let's discuss the real problems first." The public website was unconnected to their core business web servers and managed by an agency. Sites do get hacked, errors do happen, no clients or providers or … are impacted. Just fix it, no shame or hiding or making a fuss about it.

Reports adapted to the context

This point is not a simple one. Typical technical reports will include a vulnerability rating according to some methodology, like the CVSS. The advantage of CVSS is that its scores will be rather consistent between similar technical risks.

The drawback of relying solely on CVSS and similar ratings is that they lack context.

To make this clear, these scorings do not consider reputational risk. To maintain your reputation as a top security aware company, even minor mistakes on the public website are a no go. There are websites that allow you to check on the security posture of domains. As a top security company you want a full green result.

The reverse also happens: a seemingly serious bug is not exploitable in the given context. One frequent cause is that the website is behind a WAF (web application firewall), yet, often testing is done on the webserver directly.

The costs of fixing my also be used to increase the priority of the fix: if it takes less time to fix the issue than to explain it, just fix it. An example of this are TLS configurations that need a cleanup. Exploits are unlikely, a higher priority is following the principle of better safe than sorry.

At some point the technical report content has to be lifted to the business level. The business owner does not need technical vulnerabilities scorings to decide on budgeting and priorities, what matters is to understand what can go wrong with his application and how likely it is.

Continuous improvement: reporting test metrics

Measure what you want to manage. Some objectives of metrics related to security testing are: to measure comprehensiveness of security testing, to follow progress in the state of security, to quantify success in training and awareness campaigns, to track elimination of discovered vulnerabilities, to monitor security incident response capabilities.

See addendum: metrics

Remediation assistance

Part of the reason why vulnerabilities show up is that the implementer or configurator did not know or understand why it is a mistake. To ensure that the issue will be fixed, and the error will not be repeated, the responsible will have to be briefed on the issue and the possible remediations.

A frequent question i: how can the identified problem by fixed?

The wrong expectation is that in the end, the tester is fixing the problem. The responsible for the problem should invest in understanding why it is a problem, and troubleshoot how the issue got incorporated, and how to avoid it. That knowledge should be distributed in the teams for improvement.

It may be wise to organize awareness training on these issues by specialized trainers for both didactical and practical reasons: security testers should not spend too much of their time providing awareness training.

From OWASP there is very good guidance on many application security issues and their fixes. That can provide a baseline for remediation.

Conclusion

Security testing is a broad and complex undertaking. For all security controls, requirements, standards, or other obligations, security testing must be used to confirm the item under test is indeed doing its job in implementing the measure.

The security analyst or similar function must decide which tests on which environments, in what way, by whom, the duration and the depth of test, will provide assurance that the security requirements are met for the type of system under test. There is a whole menu to choose from.

Not everything is going to be tested each and every time something has changed. A suggestion is to work with a heat map: which subsystems have undergone which type of testing? Just charting these may be an eye-opener. That is a topic for another document.