When I teach exploratory testing to people, I always start with an application they don’t know. Let’s face the facts: Exploring something new is a different activity than exploring an area you are familiar with. We don’t really know what we are doing unless we have a longer course or look at the context of the application we use.
Two of my favourite targets for testing are the ones I use right now. They are both small so it is easy to learn them and complete the testing activity. Aside from the variety of targets I offer, I also teach using them as targets.
The first is a web app that displays input fields and outputs to a user interface. The second is code that we write as we explore the problem and decide how to implement it.
As a tester with both targets, I’ve had the pleasure to observe tens of testers working together on the testing problem.
1. Begin with input filtering tests
A lot of testers were taught negative testing by someone. They start by avoiding input fields that are UI or API-level. This is a relevant test but it only works if you do it first.
- Find out what positive applications of the application look like
- Be aware that an effort is made to correct errors
- I want to specifically test the input filtering once it is in place
We can see that the code-oriented activity only allows us to understand and appreciate input filtering when we explicitly state our intention of error handling and input filtering. Both activities require us to be able to recognize and understand what is wrong input before we can identify the baseline of correct input.
Many testers don’t understand how it might work, and who would care? Don’t.
2. Only one sunny day
As testers were trained about negative tests, they also learned about sunny day scenarios. It appears that testers believe there is only one sunny day scenario. In reality, there are many and there is a lot of variation. There are many things to explore, even if we don’t use the correct inputs. You can change the order of things. You can change what you type. You can change the time between your actions. It is possible to vary the way we observe.
There are many positive aspects to our sunny day scenario, and we should start by looking for them. It is more important to address problems that arise in sunny day situations.
People stop exploring because they imagine one sunny day.
3. Begin with the long and complex
Many people create complex or lengthy scenarios to ground exploration to something that is relevant. It’s great for exploring but it can also be used as an anchor to learning.
- We must face the scenario, even if it means we have to be blind for the reality of the application
- We think we made it through this scenario, no matter what happened.
People are more adept at tracking small variations than large ones, according to me. It is easy to have an idea of the scenario, but taking notes and naming smaller items will yield better results.
It can also lead to delays in finding the basic information, if you set up something complicated that is difficult to understand. People are often slow to understand complex problems and take too long to fix them later. This is something I have seen repeatedly.
This begs the question: Does speed of feedback really matter? It should be obvious that the fix will have to be repeated. However, knowing the problem earlier can help you get motivated to fix it. It’s better to fix it later than never.
4. Concentrate on the thing you don’t have
This is often a sign that you are not thinking in terms of what is yours, architecture-wise. Who will help you fix something that doesn’t work? Your team can discuss how they use third-party services and then change to another one. You don’t have to always test what you rely upon just because you can.
It is easier to make sense of information that we provide to our teams if we consider the feedback we want to respond on and can react to. It all has to work together. However, if we know who provides certain functions, we can have conversations about how to react to feedback that we might otherwise miss.
5. Too much emphasis on usability
Usability is crucial. You are likely to have some ideas as you explore a new application or domain. Sometimes, we push these ideas too far to the center that we don’t see the results we want.
This is usually a sign of the “even the broken clock is right twice daily” syndrome. In this scenario, any test results are good and not the whole purpose of testing. It’s great to be able say I found 10 bugs. Sometimes, it can also make us forget which bugs those 10 bugs were.
Sometimes, it is a good idea to delay reporting bugs, especially usability bugs. This allows you, the tester to ask yourself if you still see the same issues after 4 hours of experience and if learning the application changes your feedback.
6. Only for explicit requirements
My pet peeve is giving up control over requirements to an outside source. Exploratory testing is actually the process of identifying requirements theories and then having discussions about these requirements. Exclusion from explicit requirements can lead to a loss in results.
Exploratory testing can uncover many important issues, including bugs of omission. These are the things we should reasonably expect to see and believe, but they don’t. These things are not always obvious to us, but we can be more creative by using our imagination and thinking about what might be.
The code-oriented exploratory test target has two dimensions. It assumes that the PO knows the truth, but the PO will be wrong if it is not.