E2E Security Testing via exploratory Testing

How do you do security testing? Do you work in silos and test the product and sign off that the changes are good to go. This type of testing is called ad hoc testing.

Ad Hoc Testing

Testing that is done in ad hoc basis is unstructured. Ad Hoc Testing only delivers outputs . The output here is reporting security issues to the development teams or other stakeholders and waving the green flag to deploy the changes in production. This can be done when security has been involved at the last time and we have a very short time to handle. And THIS IS A BAD SMELLLLL. We should not encourage ad hoc testing. Its time to shift to exploratory testing.

Ad Hoc Testing leads to bias testing. In this testing, engineers would perform a sanity check and sign off ( sign off is to say that all are good and the changes can be deployed in production) due to various biases and factors. 

Exploratory Testing

In exploratory testing, the goal of security changes from finding security issues to fixing as much as security issues in less time with higher collaboration and feedback. Two of the most important factors in security testing are test cases and automatability which drives the testing outcome. The other goals that exploratory testing should be followed are

1. Collaboration: Test cases should be tracked and noted down.This help in improving visibility, collaboration and confidence in security testing.

2. Favor automated test cases: Incentivise repeatable test cases rather than finding cool security issues.

3. Communication: Findings should have reproducible POC Video ,steps and helping artifacts such that devs can easily understand and fix them. Giving reasons on why the finding should be prioritized.

4. Security Debts: Managing unfixed security issues as security debts to avoid security singularity.

5. Influence: Drive fixes to have security influence in code and production environment 

Exploratory testing will help us in driving collaboration, automating test cases thereby avoiding retesting, improving communication. 

We can leverage Critical findings to convince leadership to invest in security. But if the time spent in finding critical findings is difficult to exploit and needs skills then it is a waste of time where we could have spent time in finding easy to find exploitable vulnerabilities which drives coverage and confidence. I would recommend incentivizing coverage, collaboration and confident security testing rather than finding difficult to exploit vulnerabilities as testing is scrutinized by time, resources and skills. I am not suggesting to entirely ignore, we could perform those types of testing to find only critical vulnerabilities which needs time and effort in thematic based testing. 

Security engineers ( even I ) comes from strong motivation to find critical issues. They are very passionate on cybersecurity and its problems. But motivation to find critical issues has less value than goals mentioned above. Passionate Security engineers have to move from motivation driven to consistency driven.

Automating Test cases

If you think about automating test cases, the major challenge that we face is matching the response of the test suite. How do you verify that the test cases have been successfully run? 200 responses? Grep some text in the response?. 

1. Some test cases are performed for slowing the system causing ddos attack. example, redos , ddos, query loading testing, batch jobs dos attacks.

2. some test cases are reflective in response example, xss, injection vulnerabilities, SSRF

3. some test cases causes programmatic error until the debugging values are allowed

collecting all the test cases provides visibility and improve the success rate of running it. It will help us in choosing the needed test cases and correct environment & configuration to run. For example for testing out ddos attacks, the system should help us in causing ddos in a less number of payloads/times, instead of running it in production where the test cases can fail.

This is why communicating the test cases in an excel sheet or other ways and shared among security engineers will help and improve testing by large factors. Other security engineers with exceptional skills can contribute to the test cases or someone would have already tested and we don't have to spend time unless there is a change. Now by automating test cases, even any small changes we do not have to re test.

Shifting Left Test cases

Writing test cases before development of code will help us in crowdsourcing among the team members fostering collaboration and getting all the security engineer’s skills delivered by one person. Having it from the start will get us leverage in delivering the outcome well and get collaborative insights.

Security Singularity

Unit of work in security testing is not finding sev0 bugs . Unit of work is about how many bugs are fixed, the fix rate. The quality of work is to achieve those goals specified in exploratory testing.

Security debt are security items that have risks/impact in the long term perspective. Security Issues that have not been fixed due to short term needs, due to low priority, resource problem , shortage of bandwidth will contribute to technological singularity which i call it as Security Singularity. Security Singularity causes the worst case of business halt due to security incidents. It is important to manage security debts to avoid the possibility of singularity event.

Fixing Security Issues to influence code and production environment

How do we fix security issues? just fixing the symptoms or fixing the root cause. Fixing root cause and having defenses have influence on the code thereby influence on the production environment. Decisions should cause influence and drive change rather than fix now and think later. Making a decision to use secure by default libraries will influence the code to be secure and reduce stream of vulnerabilities in the production environment.