with Security, sometimes Sacrifies need to be made.

News Flash! Testers Say More Testing is Needed.

There are two organizations which have provided me with formalized training to perform security testing; initially it was through the SANS Institute where I completed a number of their classes and passed a number of certifications and then more recently through the Black Box Software Testing (BBST) course from the Florida Institute of Technology with Dr. Kaner.  SANS helps by providing a concise method for quickly getting up to speed in an area (mostly through the learning “by drinking from the fire hose” at a conference and following up later) but BBST provides a different perspective on what testing is and the value of a product.

Software Testing and Security Testing are similar enough to be confused.  That being said, Security Testers can learn a lot about testing from Software Testers.  Software Testers can also learn a lot from Security Testing.  BBST provides a test technique as well as understanding how to report bugs, and the cognitive biases of a tester.  Software Testers can learn how to determine if a flaw can be used to accomplish something besides crashing a program or displaying text incorrectly.

With that in mind, recently there have been a few conference topics discussing what needs to change in the security testing community.  They are advocating more penetration testing and moving away from vendor provided solutions.  A tester saying that more testing is needed is like a policy person saying that more policies are needed, or a requirements person saying that more requirements are needed.  To be fair, they also called for better testing in terms of what and how systems are tested (requiring and holding Penetration Testers to the Penetration Testing Execution Standard) and better reporting to ensure that the items identified will be fixed.

Unfortunately this is not a small issue; most organizations cannot abandon their compliance mission especially government organizations.  For those that fall under the Federal Information Security Management Act (FISMA), NIST Special Publication (SP) 8005-53; Recommended Security Controls for Federal Information Systems and Organizations, has the Vulnerability Testing security control (RA-5), while Penetration Testing is a security control enhancement (RA-5(9)) it is not required for any overall system security impact level.  There are a couple of issues with this: 1) organizations do not typically require security controls to be implemented beyond what is required from SP 800-53, and 2) the “Penetration Testing” that is listed as part of RA-5(9) is in reality just Vulnerability Validation (the supplemental guidance to CA-2(2) supports this claim and refers to Red Team although the definition listed in SP800-53: Appendix B runs counter to this interpretation).

There are a few other problems with simply relying or implementing an expanded testing program.

  1. Although a program may identify problems it will identify them late in a system’s lifecycle after most of the design and implementation has been completed.  Problems identified will be expensive to fix and cost more than if the problem was identified early in the life cycle.  Instead of spending time and resources acquiring additional highly skilled test resources, it may be more effective to invest in developer training, implementing a static code analysis program, having requirements staff more clearly define requirements/limit scope creep.
  2. If we assume that it was possible to construct a set of all of the system’s vulnerabilities; each vulnerability of this set would has probability of discovery which would be dependent on the tester’s skill level, the tester’s cognitive biases, the tester’s available resources and the time available.  Any one of these variables would affect the likelihood of discovering a vulnerability.  Even after a vulnerability has been discovered it is up to the organization to correct it.  The probability of the vulnerability being fixed depends on several factors, including but not limited to the estimated business impact and cost of correction verses cost of exploitation.  They may decide not to correct it if they do not think that there will be a business impact or the impact will be less than the cost of correcting it.  Even then, at the end of the day, your adversaries may not have the same priorities as you so the vulnerability that was corrected was not the one they used.  Another point to consider is that the skill level, cognitive biases, available resource, and time available will be different for a penetration tester/ethical hacker as opposed to an actual adversary.

There are many ways to accomplish a reduction in risk; having more testers may not be the best option for all organizations.  Some smaller organizations, no matter how committed to the idea of a strong testing program, may not be able to afford to have proficient security testers on staff.

We all have our own inherit cognitive biases (and as testers we also need to recognize that we have experimenter biases) and if you support the Sapir-Whorf Hypothesis (see Linguistic Relativity) then the language we use to describe our world also shapes (or controls) our responses.  Saying that increased and better testing is one thing; but entirely relying on testing to solve your issues may not work out well in the long run.  Testing is important, and thorough testing is even more important, but testing is only part of the process.

Comments are currently closed.