Dr. Edsger Dijkstra – one of the luminaries in the firmament of computer science – once said: “Program testing can be used to show the presence of bugs, but never to show their absence!”
The proper role of testing in the software development life cycle is to validate the soundness of the engineering process that produced the software.
Testing – as a component of the quality process – follows its own development process often, and properly, in parallel and coordinated with the development of the system. That is how the “V” described in an earlier post functions irrespective of the actual defined development process in use.
Tests are planned and designed to provide coverage of the system – not only the externally visible interaction with the user, but also coverage of the structure of the code itself. But that is not all there is.
What the system is supposed to do … usually called the system requirements … is established by the customer (internal or external). This begins life as a more or less vague idea in the mind of the customer. Any appreciably interesting system behavior is going to take more than one person to create. Which means that things need to be written down. Activities need to be coordinated.
For example, there is no way to “test” a statement of requirements … the only way to assure the quality (correctness and completeness) of the requirements is to review them with the customer. So, the statement of requirements needs to be written to be communicated to the customer and then reviewed with them.
From the 30,000 ft level, none of the early effort in a project will be “testable.” The tests themselves are not, strictly speaking, tested, either.
- How do you know that the test you are planning to run tests that part of the system that you were planning to test?
- How do you know that your test is not actually testing a completely different part of the system than the part you think you are testing?
- What constitutes a correct set of pre-conditions (what must be true before running the test so that the test has validity) before executing a test? How do you validate those?
None of this is meant to disparage software testing – it is a necessary activity in any development process. The point is that testing is not all there is in a software quality process.
I can hear your unasked question: how much does all that cost? How could it be afforded? We will take up those questions in a later post.