1.3 What is a “Test”

Tests are a measure of the system’s conformation with user expectations.

This means that testing is a form of specification writing. The generation of a test is not an act unto itself, it is a part of the defining the scope of the system.

Notice that I say "the generation of a test": Tests are not an action that is performed, they are a definition of a set of steps.

Testing is a guide for action, not the action itself.

Tests are Consistent

In order for the Test to be meaningful it has to be consistently reproducible. In many ways the Test is like a laboratory definition of the set of steps that make up a scientific experiment. When we don't achieve the anticipated result, we have learned something interesting. What is important is that the steps produce consistent results. If the results are inconsistent (different things happen at random) it is not possible to derive any meaning from them. It is the reproducibility of the test that makes it meaningful: when I take certain steps, certain events occur.

In order for a test to be reproducible, the exact steps to reproduce the results need to be documented, in excruciating detail. By writing the steps down, you are confident that no steps will be missed. If a step is missed, then the test is invalid because you have not undertaken all the actions necessary to reproduce the results.

Once upon a time, a colleague was complimenting (ok, actually he was teasing) me on my detailed configuration instructions for a complex configuration that I had messed up. There had been some tension in the office because I had rewritten someone else's instructions to make them simpler. My joked that I had made it so bonehead simple nobody could screw it up no matter how stupid they were; it occured to me at that moment: "You have to make instructions simple, otherwise stupid people won't understand them, and smart people will find a way to misinterpret them."

Tests are Training Material

This is of huge benefit to the the New Guy. Coming in cold to any system is difficult, under these conditions, The New Guy has access to a significant amount of documentation, regardless of what his role is:

Users: acting as a tester will give a very detailed look at all of the usages they are expected to encounter. In this case, it becomes a sort of user manual.

Documentation Author: the tests represent detailed instructions for how to use the system. The new document writer can use the test list as a quick boost to writing the instructions in a distributable manner.

Developer: developers want to look for samples of code when learning a new system. Done correctly, tests act as an in-house library of code samples.

Closely related to New Guy, is Future Self. Tests are a way of communicating with ourselves when designing new features, or redesigning existing ones. By writing the expectations out in a detailed fashion, all parties have a written contract as to what their expectations are. Future Self also has a way to follow up with the rationale that prompted doing something the way it was done in the first place. While the expectations may change, we have clearly defined what they are at this point.

In the end a Test is simply a form of garnering Feedback. The results of tells the developers that Testing is about garnering feedback. The purpose of feedback is so we can understand where we need to take action.

Therefore testing is a guide for action, not the action itself.

Test Driven Development and Intensive Fast Failure

Dr. Jack Matson - Intensive Fast Failure - wrote “Innovate or Die”