I’ve never seen more useless explanations and examples in software testing than I’ve ever seen about pairwise testing. For me whenever I tried to understand this test technique I never understood it better even after reading articles after articles. However, after I finally understood it I was given a powerful technique to use in my testing processes. So if you also have a hard time understanding the pairwise testing technique, lean back, and let me explain it to you.
- What is Pairwise Testing?
- Introducing an Example for Explanation
- What is the Motivation for Pairwise Testing?
- Pairwise Testing in Practice
- A Concrete Example
What is Pairwise Testing?
“Pairwise testing is a technique we can use to try to control this combinatorial explosion when we have unconstrained options. Conceptually, in pairwise testing we make sure that each option is represented in at least one test configuration, that each possible pair of options is represented in at least one test configuration, and that each option and pair of options are represented about equally as a percentage of the total configurations. Not all possible higher-order combinations, such as triples, quadruples, quintuples, and so forth, will be covered.” /Rex Black, Advanced Software Testing Vol. 1, 4.2.28./
Introducing an Example for Explanation
| Value 1 | Value 2 | Value 3 | Value 4 | |
| Dropdown A | A1 | A2 | A3 | A4 |
| Dropdown B | B1 | B2 | B3 | B4 |
| Dropdown C | B1 | C2 | C3 |
We have 3 dropdowns: two have 4 while one has 3 options.
If you would like to test all the combinations it is 4*4*3 = 48 test cases. Why would you do that? Because you want to check that all of these combinations are working well. But is it really necessary?
What is the Motivation for Pairwise Testing?
The theory behind pairwise testing is that it is more likely for an error to occur when using certain values or certain pairs of the options, but less likely on a higher level.
So the probability of issues occurring regarding our examples using a certain value of a certain dropdown is the highest. For example selecting B2 will crush the system.
Then the second most probable way for bugs to occur is in pairs. For example selecting A2 and B3 at the same time will crush the system. In this example A2 won’t crush the system if we choose it with B1 and C3, but for some reason choosing the A2-B3 pair will crush the system.
And the higher in order we go the less likely that the combination causes the problem.
So in a test case where we use A1, B1 and C1 and the system crashes, it is most likely that the problem is with either A1 or B1 or C1.Second most likely is that it is with the A1-B1, A1-C1 or B1-C1 pairs. It is least likely that the bug occurred due to using the A1-B1-C1 triple as a whole..
Of course as we have 3 dropdowns in the example we will always need to select one value from each, but this does not mean that we are focusing on triples.
Pairwise Testing in Practice
First, we want to ensure that we will test all the dropdown options. What does this mean?
With a test case including A1, B1 and C1 we have already covered 3 of them. In our example we can cover all the input values in four test cases:
- TC1 – A1;B1;C1
- TC2 – A2;B2;C2
- TC3 – A3;B3;C3
- TC4 – A4;B4;C?
In TC4 it doesn’t matter which C value is chosen as we have already covered all the Cs.
On the second layer we have to cover all possible pairs. So let’s take A1. A1 can be in pairs with any value from dropdown B and from dropdown C.
With the test TC1 we have already covered 2 pairs of A1: A1-B1 and A1-C1. In our example A1 can be in pairs with 7 values, as can any other A and B values, while C values can be in pairs with 8. We want to ensure that all of these pairs will occur in our test cases.
Now, let’s take a look at TC1. We have already noticed that we are testing two pairs A1-B1 and A1-C1, but we are also testing the B1-C1 pair. So in this setup we are able to test three pairs in one test case. As not all of the dropdowns have the same amount of options, as we proceed we will duplicate certain pairs.
However, it is the fact that we have three dropdowns which leads us to always include three values, not that we want to test triples. Our focus on the other hand is on the doubles within the test cases.
There are certain techniques on how to make a table with all the pairs (see more: Rex Black, Advanced Software Testing Vol. 1), and also tools that will help you to do so. For more complex cases I recommend the following website: https://pairwise.teremokgames.com/
A Concrete Example
Here I will present a very simple method. It might not be applicable with more factors and values, but it is a good way to understand the concept.
As we will have 3 options in every test case, let’s create a table with three columns. For clarity we will add plus one column at the beginning for the test case number.
First, fill up the table with the A and B pairs. The easiest way to do so is to put A1 and then the B options, A2 and the B options and so on:
| TC1 | A1 | B1 | |
| TC2 | A1 | B2 | |
| TC3 | A1 | B3 | |
| TC4 | A1 | B4 | |
| TC5 | A2 | B1 | |
| TC6 | A2 | B2 | |
| TC7 | A2 | B3 | |
| TC8 | A2 | B4 | |
| TC9 | A3 | B1 | |
| TC10 | A3 | B2 | |
| TC11 | A3 | B3 | |
| TC12 | A3 | B4 | |
| TC13 | A4 | B1 | |
| TC14 | A4 | B2 | |
| TC15 | A4 | B3 | |
| TC16 | A4 | B4 |
Now let’s include the C1 option to the A options. While doing it we have to pay attention to always choose a row with a different B option. In this case we won’t duplicate the C1-B1 pair:
| TC1 | A1 | B1 | C1 |
| TC2 | A1 | B2 | |
| TC3 | A1 | B3 | |
| TC4 | A1 | B4 | |
| TC5 | A2 | B1 | |
| TC6 | A2 | B2 | C1 |
| TC7 | A2 | B3 | |
| TC8 | A2 | B4 | |
| TC9 | A3 | B1 | |
| TC10 | A3 | B2 | |
| TC11 | A3 | B3 | C1 |
| TC12 | A3 | B4 | |
| TC13 | A4 | B1 | |
| TC14 | A4 | B2 | |
| TC15 | A4 | B3 | |
| TC16 | A4 | B4 | C1 |
Let’s do the same thing for C2 and C3 as well:
| TC1 | A1 | B1 | C1 |
| TC2 | A1 | B2 | C2 |
| TC3 | A1 | B3 | C3 |
| TC4 | A1 | B4 | |
| TC5 | A2 | B1 | C2 |
| TC6 | A2 | B2 | C1 |
| TC7 | A2 | B3 | |
| TC8 | A2 | B4 | C3 |
| TC9 | A3 | B1 | C3 |
| TC10 | A3 | B2 | |
| TC11 | A3 | B3 | C1 |
| TC12 | A3 | B4 | C2 |
| TC13 | A4 | B1 | |
| TC14 | A4 | B2 | C3 |
| TC15 | A4 | B3 | C2 |
| TC16 | A4 | B4 | C1 |
With the table above we covered all the pairs in 16 test cases. Let’s take a look, and check if we have A1-B1, A1-B2, A1-B3, A1-B4, A1-C1, A1-C2, etc. somewhere in the test cases.
As we had only three options for C we have a few (4) empty cells. We can either leave it blank, fill it randomly with C options, or mark it as tester choice so the tester executing it can decide.
I hope this explanation helps you understand pairwise testing more.
For a more detailed explanation I highly recommend reading the pairwise testing part (4.2.28) from Advanced Software Testing Vol. 1 from Rex Black.