What to know about Exhaustive Testing?

What is Exhaustive Testing?

Exhaustive testing, which is also known as complete testing, occurs when all the testers in your team are exhausted and when all the planned tests have been executed. It is a quality assurance testing technique in which all scenarios or data is tested for testing. In a more understandable way, exhaustive testing means ensuring there are no undiscovered faults at the end of the test phase. Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. As testers, we often say “well I just never have enough time for testing”. Even if you had all the time in this world, you still wouldn’t have enough time to test all the possible input combinations and output combinations.

Strategies for exhaustive testing

We know that one of the 7 testing principles of software testing says that exhaustive testing is not possible. Now the question which lies is that: as exhaustive testing is not possible how we are supposed to get enough coverage to tell that the application under test is tending to not having any defects?

Instead of trying to test everything with what we tend to do and that we want to apply some helpful approaches, we use risk and priorities to focus testing efforts. We may do techniques such as pairwise testing to reduce the number of combinations, or things like cause-effect graphing and do more of a logic-based approach to optimize our testing. Here are basic and fundamental strategies that one can definitely include in their test plan:

Impact on Business: This not the only one but is one of the most important parameters to understand the risk. Testers should question that if a certain functionality of a module being tested fails how much risk it will bring to the entire module. Testers should constantly work with the Business Analyst of the module to understand the threat brought by the failure of each functionality so that it can be prioritized high up in the testing.

Probability of failure: This parameter also plays a major role in understanding the health of a functionality. Some functionality can have a high probability of failure compared to other functionalities, for example, let’s consider an e-commerce site which will have one functionality of logging in as a user and other to determine the users browsing history and correctly give the user more options to look at(cross sell/up sell). As a tester, we can identify that the second functionality has a higher probability of breaking because it contains a lot of business algorithms to be implemented compared to a login page which might have less probability of failure.

Regression: Testers should look at this parameter by identifying functionalities which have to be regressed more often when there is a change or addition to certain areas of the AUT. Correct impact analysis should be done to determine the areas correctly. Impact analysis should always involve developers, QA, and the Business Analyst team to get a correct understanding of the impacted areas.

Recovery: If a functionality breaks how fast can it be recovered or the recovery time taken. QA personnel can get the inputs from developers on the recovery time for each functionality of the application under test.

Why Exhaustive Testing is impractical and impossible?

It is not possible to perform complete testing or exhaustive testing. For most of the systems, it is near impossible because of the following reasons:

The domain of possible inputs of a program is too large to be completely used in testing a system. There are both valid inputs and invalid inputs.
The program may have a large number of states. There may be timing constraints on the inputs, that is, an input may be valid at a certain time and invalid at other times. An input value which is valid but is not properly timed is called an inopportune input. The input domain of a system can be very large to be completely used in testing a program.
The design issues may be too complex to completely test. The design may have included implicit design decisions and assumptions. For example, a programmer may use a global variable or a static variable to control program execution.
It may not be possible to create all possible execution environments of the system. This becomes more significant when the behavior of the software system depends on the real, outside world, such as weather, temperature, altitude, pressure, and so on.

Exhaustive Testing Examples

The IE Tools > Advanced Options window

53 binary conditions

1 condition with 3 options

1 condition with 4 options

2⁵³ = 9,007,199,254,740,992

x 12

= 108,086,391,056,891,904 possible combinations of conditions

At one second per test execution:

108,086,391,056,891,904 / 360 = 300,239,975,158,033.067 hours (12,509,998,964,918.04 days or 34,273,969,766.9 years) to test all possible combinations.

More details are available here for the deep learning about this technique.