While experience-based testing provides benefits by efficiently finding defects that other test techniques may miss and serving as a check on the completeness of those techniques, it provides some challenges for test management. Our Test Manager is aware of the challenges as well as the benefits of the experience-based techniques, particularly exploratory testing. It is difficult to determine the coverage attained during such testing, given the typical light-weight logging and minimal advanced preparation of tests. Reproducibility of the test results requires particular management attention, especially when multiple testers are involved.
One way to manage experience-based testing, especially exploratory testing, is to break the testing work into small, 30 to 120 minute periods sometimes called test sessions. This timeboxing limits and focuses the work to be done in a session and provides a level of monitoring and scheduling. Each session covers a test charter, which is communicated in writing or verbally to the testers by our test manager. The test charter gives the test condition(s) to be covered in the test session, which further helps to maintain focus and prevent overlap if multiple people are carrying out exploratory testing simultaneously.
Another technique to manage experience-based testing is by integrating such self-directed and spontaneous testing into more traditional pre-designed testing sessions. For example, testers can be given permission (and allocated time) to explore beyond the explicit steps, inputs, and expected results in their pre-defined tests. Testers will also be assigned such self-directed testing sessions as part of their daily testing, before, during, or after a day of running pre-defined tests. If such testing sessions identify defects or interesting areas for future testing, the pre-defined tests will be updated.
At the beginning of the exploratory session, our testers ascertain and perform the necessary set up tasks for the tests. During the session, our testers learn about the application being tested, designs and execute tests according to the technique being applied and what has been learned about the application, investigate any defects, and capture the results of the test in a log. (If repeatability of the tests is required, our testers will also log the test inputs, actions, and events.) After the session, a debriefing may occur, which sets the direction for subsequent sessions.