Exploratory Testing as a Way to Find Non-Obvious Problems

One particular bug we encountered stayed with us for weeks. The user’s cart would spontaneously clear out. Of course, there were absolutely no issues discovered during any of the automated tests.

Well, apparently there was one specific combination of events that caused that. Someone put an item into the cart, opened another tab for a couple of minutes, went back, and tried to increase its quantity. Nobody ever thought of writing a test case for that.

And that particular episode got me thinking. While scripted testing is great at proving that your software works correctly according to the specification, it’s really bad at detecting anything else.

The Limitation of Scripted Tests

Scripted testing, whether done manually by filling in checklists or automatically through pre-written code, is an exercise in confirmation. Put input X, press button Y, verify output Z. If all goes well, continue to step 4.

You absolutely need this level of validation. But it’s only going to catch the problems someone already knew could occur. If your test cases are derived from the documentation of the product, the testing would be limited to validating its correctness.

But real-life users couldn’t care less about what’s written in your specifications document. They may copy-paste a phone number to an email field, spam the “submit” button with a rage-induced fury, and access your web application through their five-year-old cracked screen Android device.

The Actual Nature of Exploratory Testing

There is this common misconception: “So, you basically click around randomly, right?” Wrong. It’s closer to being a detective at the crime scene than anything else. You don’t come in with a set of 50 pre-defined steps to perform. You examine what’s there, see something unusual, investigate it, and so on.

In exploratory testing, the process of investigating the software under test becomes intertwined with building hypotheses of what could potentially go wrong with the product and immediately verifying those hypotheses. For instance, a slightly lagging page load, unusual UI animation, or slow response from the API are all potential signs of something going awry.

There are other, less obvious, ones as well. Yes, exploratory testing requires experience. Junior testers are simply less likely to come across these signs compared to seasoned developers who’ve seen it all before.

Examples of Detected Issues

Let me provide some examples of bugs discovered through exploratory testing from personal experience and that of my colleagues:

• Special combinations of edge cases. The name input box allows for 255 characters. But what if the user copies-pastes 255 emojis there, tries to submit the form, and loses their internet connection? One of the websites I tested behaved as though the input box was blank.
• UI problems at certain resolutions. An example from our development team: a modal dialog worked perfectly fine on a desktop browser. Try to open it on a 768 pixels wide tablet in the landscape orientation, and you’d be able to scroll the page right behind the dialog, with the navigation panel blocking its “close” button.
• Timing problems. By clicking through the website or application like an actual person, exploring its functionalities, switching windows and tabs, canceling an action mid-way and performing back-and-forth operations you can easily trigger race conditions in the code.
• Module conflicts. I have personally discovered one issue with our internal website where a change of the billing address triggered automatic clearing of notification preferences. Both functionality used a database update, which nobody realized was a conflict.

Two Perspectives Required

Effective exploratory testing requires two kinds of perspectives to be kept in mind at the same time. You should consider the possibility of the user behaving illogically. Someone inexperienced, confused, impatient, and not necessarily following instructions in the way they are intended.

The other side involves approaching the application with an adversarial attitude towards its workings. You should actively seek potential logical weaknesses in the product.

Why Can’t We Replace Testing Completely With Exploratory Testing?

To be honest, there are several reasons why this isn’t the most effective way. One of them is that exploratory testing doesn’t scale well. Unlike automation, it’s difficult to do in bulk and leave to run unattended at night. It also doesn’t produce easily measurable metrics. Finally, the effectiveness of this testing approach largely depends on the individual tester themselves.

How to Find the Right Balance?

An efficient quality assurance process requires a proper balance between two types of testing mentioned above.

Automated test cases are necessary to ensure proper functioning of the application. They guarantee that the system won’t break during future iterations and updates. In other words, these tests serve as your foundation.

Exploratory testing acts as a complement to automated ones by discovering problems you hadn’t considered. It’s an extra layer to your testing strategy. Neither of them can be omitted. At least, not for long. At Swan Software Solutions, we know this very well.

If you need quality assurance or other technology services, schedule a free assessment to find out how Swan Software Solutions can help.