Higher Strength Combinatorial Interaction Testing

This page is supplementary to the paper entitled "Efficiency and Early Fault Detection with Lower and Higher Strength Combinatorial Interaction Testing" ([PDF] [BIB]), which is currently to appear in the proceedings of the ESEC/FSE 2013.

Abstract

Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. However, in order to be practically applicable, it must be able to cater for soft and hard real-world constraints and should, ideally, report a test priority order that maximises earliest fault detection. We show that we can achieve the highest strength CIT in 5.65 minutes on average. This was previously thought to be too computationally expensive to be feasible. Furthermore, we show that higher strength suites find more faults, while prioritisations using lower strengths are no worse at achieving early fault revelation.

Experimental Setup

Subject Programs

We use five Unix utilities from the Software Infrastructure Repository (SIR), maintained by University of Nebraska Lincoln: flex, grep, gzip, sed, and make.

Test Suites

The test suites we use are subsets of those provided by SIR. It can be downloaded from here. For each of the subjects we provide:

  1. An annotated TSL file (from SIR) : _v0_#_small.tsl
    Description: The v0_# matches the associated TSL name from SIR for that subject. The leftmost column in this file is a mapping to the covering array values given as sequential numbers starting at 0. For instance, the first parameter if binary will be 0,1 and the next parameter's values will start at 2. The single and error parameters (and any other parameters that we did not include in our model) are not numbered and not used in our experiments.
  2. An input model file for CASA: _v0_small.model
    Description: The first row is the strength (t) and can be changed to 3,4 etc. to generate higher strength arrays. The second row is the number of factors (TSL parameters in the model). The last row lists the number of values (TSL choices) for each of the parameters. This model matches the associated TSL file.
  3. The constraint file in CASA input format: _v0_small.constraints
    Description: The first row of this file is the number of constraints. For each of the constraints there are two rows. The first row gives the arity of the constraint (i.e. how many values will be included in this constraint). The next row is list of the constraints between values. The - signs indicate excludes. For instance, a row with a 2, followed by a row that has -10 -12 encodes the binary constraint that values 10 and 12 cannot appear together in a test. Please see the CASA tool for full documentation.

Combinatorial Interaction Testing Tool

We use CASA (Covering Arrays by Simulated Annealing) tool to generate combinatorial interaction test suites from the TSL files.

Full Results

This section contains the full APCC data for all experiments and plots for APCC and fault detection.

Subject Generation Strength Prioritsation Strengths Subject Generation Strength Prioritsation Strengths
flex 2 2, 3, 4, 5 gzip 2 2, 3, 4, 5
flex 3 2, 3, 4, 5 gzip 3 2, 3, 4, 5
flex 4 2, 3, 4, 5 gzip 4 2, 3, 4, 5
flex 5 2, 3, 4, 5 gzip 5 2, 3, 4, 5
grep 2 2, 3, 4, 5 sed 2 2, 3, 4, 5
grep 3 2, 3, 4, 5 sed 3 2, 3, 4, 5
grep 4 2, 3, 4, 5 sed 4 2, 3, 4, 5
grep 5 2, 3, 4, 5 sed 5 2, 3, 4, 5

flex

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.851 71.707 60.886 51.033
5 97.674 89.340 78.722 67.791 57.518
6 100.000 93.163 83.403 72.852 62.623
7 100.000 94.206 85.516 75.762 66.013
8 100.000 95.017 87.113 77.908 68.466
9 100.000 95.829 88.849 80.407 71.487
10 100.000 96.408 90.086 82.192 73.659
11 100.000 96.987 91.240 83.759 75.468
12 100.000 97.451 92.394 85.598 77.854
13 100.000 97.798 93.160 86.739 79.287
14 100.000 98.030 93.760 87.730 80.618
15 100.000 98.262 94.268 88.491 81.588
16 100.000 98.494 94.831 89.396 82.795
17 100.000 98.725 95.458 90.470 84.280
18 100.000 98.957 95.994 91.305 85.375
19 100.000 99.189 96.594 92.307 86.742
20 100.000 99.305 96.843 92.677 87.212
21 100.000 99.421 97.175 93.258 88.023
22 100.000 99.537 97.434 93.656 88.545
23 100.000 99.652 97.711 94.097 89.130
24 100.000 99.768 98.006 94.592 89.818
25 100.000 99.884 98.274 95.012 90.372
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.271 70.747 59.812 50.024
5 97.674 89.108 78.418 67.553 57.408
6 100.000 93.048 83.310 72.881 62.799
7 100.000 93.975 85.240 75.594 66.014
8 100.000 94.670 86.753 77.807 68.727
9 100.000 95.481 88.433 80.170 71.538
10 100.000 96.176 89.901 82.268 74.067
11 100.000 96.871 91.341 84.300 76.498
12 100.000 97.219 92.218 85.721 78.377
13 100.000 97.567 93.021 86.959 79.971
14 100.000 97.914 93.806 88.142 81.462
15 100.000 98.262 94.581 89.308 82.934
16 100.000 98.610 95.320 90.379 84.256
17 100.000 98.841 95.855 91.219 85.365
18 100.000 99.073 96.381 92.021 86.389
19 100.000 99.189 96.705 92.589 87.195
20 100.000 99.305 97.009 93.109 87.925
21 100.000 99.421 97.268 93.507 88.448
22 100.000 99.537 97.526 93.899 88.950
23 100.000 99.652 97.775 94.271 89.426
24 100.000 99.768 98.025 94.640 89.896
25 100.000 99.884 98.274 95.010 90.366
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 81.808 69.934 58.848 49.062
5 97.674 88.528 77.375 66.284 56.108
6 97.674 90.151 80.504 70.398 60.699
7 97.674 91.078 82.424 73.093 63.894
8 97.674 91.889 84.187 75.652 66.997
9 97.674 92.352 85.267 77.317 69.125
10 100.000 96.060 89.670 81.958 73.716
11 100.000 96.408 90.538 83.356 75.562
12 100.000 97.103 91.978 85.384 77.980
13 100.000 97.451 92.791 86.642 79.600
14 100.000 97.798 93.557 87.793 81.063
15 100.000 98.146 94.341 88.982 82.571
16 100.000 98.494 95.126 90.168 84.073
17 100.000 98.725 95.671 91.019 85.181
18 100.000 98.957 96.178 91.780 86.151
19 100.000 99.189 96.705 92.582 87.175
20 100.000 99.305 97.000 93.078 87.863
21 100.000 99.421 97.268 93.498 88.418
22 100.000 99.537 97.526 93.896 88.940
23 100.000 99.652 97.775 94.271 89.426
24 100.000 99.768 98.025 94.642 89.902
25 100.000 99.884 98.274 95.012 90.372
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.954 68.993 58.624 49.270
5 93.023 84.125 74.328 64.519 55.269
6 95.349 87.717 78.685 69.298 60.196
7 97.674 91.194 82.766 73.665 64.635
8 97.674 92.236 84.870 76.556 68.001
9 97.674 92.816 86.135 78.421 70.313
10 97.674 93.395 87.409 80.302 72.642
11 100.000 96.640 90.972 83.914 76.172
12 100.000 97.335 92.394 85.900 78.531
13 100.000 97.683 93.178 87.092 80.054
14 100.000 98.030 93.963 88.282 81.565
15 100.000 98.262 94.508 89.137 82.686
16 100.000 98.378 94.858 89.773 83.600
17 100.000 98.610 95.403 90.628 84.721
18 100.000 98.725 95.708 91.150 85.456
19 100.000 98.957 96.197 91.860 86.337
20 100.000 99.189 96.741 92.705 87.423
21 100.000 99.421 97.268 93.507 88.448
22 100.000 99.537 97.517 93.879 88.924
23 100.000 99.652 97.766 94.251 89.400
24 100.000 99.768 98.015 94.621 89.870
25 100.000 99.884 98.265 94.990 90.340
26 100.000 100.000 98.523 95.382 90.842

flex

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 86.047 69.757 55.183 43.241 33.810
3 90.698 77.520 64.405 52.735 42.856
4 95.349 84.820 72.851 61.352 51.071
5 100.000 91.309 79.886 68.304 57.631
6 100.000 93.048 83.283 72.814 62.698
7 100.000 94.322 85.821 76.250 66.635
8 100.000 95.249 87.723 78.901 69.762
9 100.000 96.060 89.357 81.147 72.387
10 100.000 96.756 90.843 83.296 75.004
11 100.000 97.103 91.609 84.434 76.427
12 100.000 97.451 92.458 85.785 78.194
13 100.000 97.798 93.252 86.997 79.738
14 100.000 98.146 93.981 88.041 81.010
15 100.000 98.378 94.609 89.104 82.459
16 100.000 98.610 95.255 90.212 83.980
17 100.000 98.841 95.818 91.119 85.194
18 100.000 98.957 96.114 91.621 85.902
19 100.000 99.073 96.372 92.019 86.424
20 100.000 99.189 96.649 92.459 87.009
21 100.000 99.305 96.917 92.877 87.558
22 100.000 99.421 97.203 93.347 88.198
23 100.000 99.537 97.471 93.772 88.770
24 100.000 99.652 97.738 94.191 89.325
25 100.000 99.768 98.052 94.746 90.125
26 100.000 99.884 98.301 95.117 90.601
27 100.000 100.000 98.569 95.537 91.156
28 100.000 100.000 98.643 95.745 91.529
29 100.000 100.000 98.726 95.972 91.924
30 100.000 100.000 98.735 96.003 91.985
31 100.000 100.000 98.800 96.180 92.294
32 100.000 100.000 98.837 96.282 92.474
33 100.000 100.000 98.911 96.494 92.860
34 100.000 100.000 98.994 96.726 93.273
35 100.000 100.000 99.086 96.980 93.715
36 100.000 100.000 99.169 97.223 94.164
37 100.000 100.000 99.178 97.250 94.215
38 100.000 100.000 99.188 97.277 94.266
39 100.000 100.000 99.197 97.306 94.320
40 100.000 100.000 99.289 97.560 94.767
41 100.000 100.000 99.400 97.875 95.334
42 100.000 100.000 99.409 97.905 95.395
43 100.000 100.000 99.492 98.136 95.799
44 100.000 100.000 99.502 98.164 95.854
45 100.000 100.000 99.511 98.191 95.905
46 100.000 100.000 99.585 98.403 96.292
47 100.000 100.000 99.631 98.538 96.538
48 100.000 100.000 99.640 98.565 96.589
49 100.000 100.000 99.732 98.817 97.027
50 100.000 100.000 99.788 98.979 97.326
51 100.000 100.000 99.852 99.159 97.647
52 100.000 100.000 99.862 99.187 97.698
53 100.000 100.000 99.898 99.289 97.877
54 100.000 100.000 99.954 99.447 98.166
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 95.349 84.936 73.064 61.614 51.341
5 97.674 88.992 78.187 67.247 57.070
6 100.000 92.816 82.895 72.377 62.284
7 100.000 94.090 85.452 75.862 66.303
8 100.000 95.017 87.390 78.600 69.560
9 100.000 95.829 89.098 81.031 72.476
10 100.000 96.524 90.566 83.128 75.005
11 100.000 97.219 91.987 85.116 77.367
12 100.000 97.451 92.624 86.204 78.862
13 100.000 97.683 93.261 87.292 80.357
14 100.000 98.030 94.046 88.476 81.849
15 100.000 98.378 94.821 89.644 83.331
16 100.000 98.494 95.172 90.290 84.276
17 100.000 98.610 95.523 90.935 85.217
18 100.000 98.841 96.040 91.718 86.220
19 100.000 98.957 96.372 92.312 87.076
20 100.000 98.957 96.511 92.687 87.721
21 100.000 99.073 96.825 93.226 88.469
22 100.000 99.073 96.945 93.559 89.056
23 100.000 99.189 97.231 94.045 89.748
24 100.000 99.189 97.332 94.324 90.237
25 100.000 99.189 97.425 94.577 90.677
26 100.000 99.189 97.517 94.836 91.139
27 100.000 99.305 97.775 95.236 91.668
28 100.000 99.305 97.849 95.451 92.061
29 100.000 99.305 97.923 95.668 92.464
30 100.000 99.421 98.181 96.066 92.987
31 100.000 99.537 98.431 96.443 93.480
32 100.000 99.652 98.680 96.813 93.950
33 100.000 99.768 98.929 97.181 94.413
34 100.000 99.884 99.178 97.549 94.879
35 100.000 100.000 99.428 97.915 95.335
36 100.000 100.000 99.483 98.071 95.617
37 100.000 100.000 99.538 98.228 95.902
38 100.000 100.000 99.594 98.385 96.183
39 100.000 100.000 99.649 98.548 96.485
40 100.000 100.000 99.695 98.682 96.735
41 100.000 100.000 99.742 98.824 97.005
42 100.000 100.000 99.788 98.957 97.250
43 100.000 100.000 99.834 99.093 97.500
44 100.000 100.000 99.871 99.199 97.694
45 100.000 100.000 99.908 99.303 97.881
46 100.000 100.000 99.917 99.330 97.932
47 100.000 100.000 99.926 99.357 97.982
48 100.000 100.000 99.935 99.385 98.037
49 100.000 100.000 99.945 99.415 98.095
50 100.000 100.000 99.954 99.442 98.146
51 100.000 100.000 99.963 99.470 98.200
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.305
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 88.372 74.739 61.885 50.692 41.300
4 93.023 82.039 70.331 59.310 49.515
5 97.674 88.644 77.605 66.592 56.452
6 97.674 90.267 80.809 70.883 61.312
7 100.000 93.975 85.276 75.688 66.165
8 100.000 94.670 86.781 77.877 68.838
9 100.000 95.249 88.064 79.782 71.205
10 100.000 95.944 89.523 81.856 73.695
11 100.000 96.524 90.741 83.602 75.818
12 100.000 97.219 92.191 85.649 78.258
13 100.000 97.451 92.800 86.673 79.658
14 100.000 97.567 93.151 87.325 80.621
15 100.000 98.030 94.138 88.755 82.373
16 100.000 98.262 94.720 89.705 83.649
17 100.000 98.262 94.849 90.065 84.285
18 100.000 98.378 95.200 90.703 85.206
19 100.000 98.610 95.744 91.558 86.327
20 100.000 98.725 96.077 92.152 87.179
21 100.000 98.841 96.381 92.677 87.923
22 100.000 98.957 96.695 93.223 88.698
23 100.000 99.189 97.194 93.962 89.635
24 100.000 99.189 97.286 94.222 90.098
25 100.000 99.189 97.388 94.502 90.589
26 100.000 99.305 97.692 95.016 91.299
27 100.000 99.305 97.785 95.269 91.739
28 100.000 99.305 97.840 95.435 92.048
29 100.000 99.305 97.905 95.621 92.389
30 100.000 99.305 97.960 95.787 92.698
31 100.000 99.305 98.015 95.952 93.004
32 100.000 99.421 98.274 96.355 93.543
33 100.000 99.421 98.329 96.518 93.848
34 100.000 99.421 98.385 96.678 94.137
35 100.000 99.421 98.440 96.836 94.426
36 100.000 99.421 98.495 96.991 94.705
37 100.000 99.537 98.745 97.365 95.188
38 100.000 99.537 98.791 97.497 95.430
39 100.000 99.537 98.837 97.628 95.669
40 100.000 99.652 99.086 97.996 96.132
41 100.000 99.768 99.335 98.364 96.595
42 100.000 99.884 99.585 98.731 97.058
43 100.000 100.000 99.834 99.099 97.521
44 100.000 100.000 99.871 99.203 97.708
45 100.000 100.000 99.908 99.305 97.888
46 100.000 100.000 99.917 99.333 97.943
47 100.000 100.000 99.926 99.362 97.997
48 100.000 100.000 99.935 99.390 98.052
49 100.000 100.000 99.945 99.417 98.102
50 100.000 100.000 99.954 99.444 98.153
51 100.000 100.000 99.963 99.471 98.204
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.306
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 79.070 62.109 48.777 38.407 30.360
3 86.047 71.958 59.356 48.629 39.714
4 90.698 79.374 68.014 57.508 48.199
5 95.349 86.327 75.898 65.511 55.852
6 95.349 87.370 78.095 68.622 59.549
7 95.349 88.181 79.830 71.124 62.579
8 95.349 88.760 81.169 73.159 65.140
9 100.000 95.133 87.898 79.628 71.093
10 100.000 95.829 89.338 81.663 73.534
11 100.000 96.408 90.557 83.410 75.656
12 100.000 96.987 91.775 85.156 77.779
13 100.000 97.451 92.781 86.637 79.619
14 100.000 97.683 93.363 87.595 80.921
15 100.000 97.798 93.714 88.240 81.862
16 100.000 98.030 94.277 89.147 83.075
17 100.000 98.030 94.397 89.481 83.662
18 100.000 98.146 94.738 90.099 84.558
19 100.000 98.262 95.071 90.692 85.407
20 100.000 98.262 95.191 91.023 85.987
21 100.000 98.494 95.754 91.921 87.171
22 100.000 98.494 95.864 92.227 87.708
23 100.000 98.494 95.975 92.533 88.246
24 100.000 98.725 96.474 93.276 89.193
25 100.000 98.957 96.972 94.014 90.127
26 100.000 98.957 97.028 94.178 90.432
27 100.000 98.957 97.083 94.339 90.730
28 100.000 99.073 97.360 94.789 91.346
29 100.000 99.073 97.415 94.947 91.631
30 100.000 99.073 97.480 95.127 91.951
31 100.000 99.073 97.591 95.422 92.454
32 100.000 99.073 97.637 95.556 92.699
33 100.000 99.189 97.905 95.982 93.275
34 100.000 99.189 97.960 96.144 93.574
35 100.000 99.305 98.228 96.567 94.140
36 100.000 99.305 98.283 96.726 94.430
37 100.000 99.421 98.532 97.100 94.913
38 100.000 99.537 98.791 97.494 95.423
39 100.000 99.537 98.837 97.628 95.669
40 100.000 99.652 99.086 97.998 96.139
41 100.000 99.768 99.335 98.366 96.603
42 100.000 99.884 99.585 98.734 97.065
43 100.000 99.884 99.622 98.838 97.253
44 100.000 100.000 99.871 99.204 97.712
45 100.000 100.000 99.908 99.306 97.891
46 100.000 100.000 99.917 99.334 97.946
47 100.000 100.000 99.926 99.363 98.001
48 100.000 100.000 99.935 99.390 98.052
49 100.000 100.000 99.945 99.417 98.103
50 100.000 100.000 99.954 99.444 98.154
51 100.000 100.000 99.963 99.471 98.204
52 100.000 100.000 99.972 99.498 98.255
53 100.000 100.000 99.982 99.525 98.306
54 100.000 100.000 99.991 99.552 98.356
55 100.000 100.000 100.000 99.579 98.407

flex

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 86.047 69.757 55.183 43.241 33.810
3 90.698 77.520 64.405 52.735 42.856
4 95.349 84.820 72.833 61.307 51.003
5 100.000 91.425 80.070 68.500 57.802
6 100.000 93.048 83.264 72.769 62.629
7 100.000 94.206 85.544 75.832 66.127
8 100.000 95.133 87.437 78.456 69.205
9 100.000 95.944 89.117 80.815 72.003
10 100.000 96.756 90.778 83.131 74.739
11 100.000 97.219 91.849 84.772 76.828
12 100.000 97.567 92.615 85.911 78.255
13 100.000 97.914 93.400 87.101 79.766
14 100.000 98.146 93.972 88.031 81.015
15 100.000 98.378 94.517 88.875 82.102
16 100.000 98.610 95.108 89.841 83.390
17 100.000 98.841 95.671 90.744 84.591
18 100.000 98.957 96.031 91.392 85.504
19 100.000 99.073 96.354 91.962 86.316
20 100.000 99.189 96.695 92.574 87.191
21 100.000 99.305 96.954 92.968 87.700
22 100.000 99.421 97.314 93.636 88.681
23 100.000 99.537 97.655 94.240 89.528
24 100.000 99.652 97.905 94.610 89.998
25 100.000 99.768 98.154 94.982 90.474
26 100.000 99.884 98.403 95.360 90.971
27 100.000 100.000 98.661 95.756 91.486
28 100.000 100.000 98.698 95.858 91.666
29 100.000 100.000 98.782 96.089 92.074
30 100.000 100.000 98.883 96.367 92.558
31 100.000 100.000 98.975 96.635 93.047
32 100.000 100.000 99.040 96.816 93.370
33 100.000 100.000 99.105 97.007 93.724
34 100.000 100.000 99.151 97.147 93.992
35 100.000 100.000 99.169 97.211 94.126
36 100.000 100.000 99.188 97.276 94.267
37 100.000 100.000 99.206 97.337 94.389
38 100.000 100.000 99.225 97.398 94.515
39 100.000 100.000 99.271 97.533 94.762
40 100.000 100.000 99.308 97.646 94.980
41 100.000 100.000 99.326 97.707 95.102
42 100.000 100.000 99.335 97.735 95.157
43 100.000 100.000 99.409 97.949 95.550
44 100.000 100.000 99.446 98.053 95.737
45 100.000 100.000 99.465 98.112 95.853
46 100.000 100.000 99.483 98.170 95.969
47 100.000 100.000 99.502 98.230 96.092
48 100.000 100.000 99.511 98.263 96.161
49 100.000 100.000 99.520 98.291 96.216
50 100.000 100.000 99.548 98.376 96.381
51 100.000 100.000 99.594 98.519 96.656
52 100.000 100.000 99.612 98.575 96.764
53 100.000 100.000 99.668 98.734 97.051
54 100.000 100.000 99.677 98.767 97.124
55 100.000 100.000 99.686 98.799 97.194
56 100.000 100.000 99.686 98.810 97.231
57 100.000 100.000 99.695 98.841 97.298
58 100.000 100.000 99.723 98.927 97.466
59 100.000 100.000 99.742 98.983 97.575
60 100.000 100.000 99.760 99.047 97.708
61 100.000 100.000 99.778 99.107 97.832
62 100.000 100.000 99.788 99.139 97.897
63 100.000 100.000 99.797 99.170 97.963
64 100.000 100.000 99.806 99.204 98.041
65 100.000 100.000 99.825 99.262 98.153
66 100.000 100.000 99.834 99.299 98.240
67 100.000 100.000 99.843 99.332 98.312
68 100.000 100.000 99.852 99.359 98.364
69 100.000 100.000 99.852 99.366 98.386
70 100.000 100.000 99.852 99.372 98.410
71 100.000 100.000 99.871 99.427 98.515
72 100.000 100.000 99.880 99.457 98.577
73 100.000 100.000 99.880 99.473 98.632
74 100.000 100.000 99.889 99.501 98.689
75 100.000 100.000 99.908 99.554 98.786
76 100.000 100.000 99.917 99.583 98.845
77 100.000 100.000 99.917 99.588 98.860
78 100.000 100.000 99.917 99.599 98.901
79 100.000 100.000 99.926 99.633 98.978
80 100.000 100.000 99.935 99.660 99.029
81 100.000 100.000 99.945 99.689 99.083
82 100.000 100.000 99.945 99.700 99.125
83 100.000 100.000 99.945 99.704 99.140
84 100.000 100.000 99.945 99.710 99.163
85 100.000 100.000 99.945 99.718 99.189
86 100.000 100.000 99.954 99.745 99.240
87 100.000 100.000 99.954 99.748 99.253
88 100.000 100.000 99.954 99.754 99.275
89 100.000 100.000 99.954 99.760 99.298
90 100.000 100.000 99.963 99.790 99.357
91 100.000 100.000 99.963 99.796 99.379
92 100.000 100.000 99.963 99.804 99.409
93 100.000 100.000 99.963 99.810 99.432
94 100.000 100.000 99.963 99.816 99.451
95 100.000 100.000 99.963 99.821 99.470
96 100.000 100.000 99.972 99.848 99.521
97 100.000 100.000 99.982 99.875 99.572
98 100.000 100.000 99.982 99.883 99.603
99 100.000 100.000 99.982 99.890 99.626
100 100.000 100.000 99.982 99.895 99.645
101 100.000 100.000 99.982 99.900 99.664
102 100.000 100.000 99.991 99.930 99.726
103 100.000 100.000 99.991 99.934 99.742
104 100.000 100.000 99.991 99.944 99.775
105 100.000 100.000 99.991 99.949 99.794
106 100.000 100.000 100.000 99.976 99.844
107 100.000 100.000 100.000 99.979 99.856
108 100.000 100.000 100.000 99.985 99.879
109 100.000 100.000 100.000 99.993 99.906
110 100.000 100.000 100.000 99.997 99.922
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.503 71.125 60.228 50.409
5 97.674 89.224 78.593 67.727 57.546
6 97.674 90.498 81.215 71.361 61.782
7 100.000 94.206 85.646 76.075 66.494
8 100.000 95.249 87.760 78.985 69.883
9 100.000 95.944 89.255 81.158 72.540
10 100.000 96.756 90.907 83.454 75.250
11 100.000 97.219 91.969 85.072 77.303
12 100.000 97.567 92.809 86.401 79.038
13 100.000 97.798 93.427 87.444 80.464
14 100.000 98.146 94.194 88.587 81.900
15 100.000 98.262 94.581 89.324 82.989
16 100.000 98.494 95.135 90.196 84.126
17 100.000 98.610 95.486 90.829 85.030
18 100.000 98.841 96.012 91.632 86.059
19 100.000 98.957 96.354 92.246 86.937
20 100.000 99.073 96.695 92.851 87.788
21 100.000 99.073 96.815 93.184 88.375
22 100.000 99.073 96.926 93.495 88.926
23 100.000 99.189 97.212 93.963 89.563
24 100.000 99.305 97.489 94.413 90.179
25 100.000 99.421 97.766 94.868 90.812
26 100.000 99.421 97.849 95.114 91.268
27 100.000 99.421 97.932 95.351 91.693
28 100.000 99.537 98.191 95.748 92.216
29 100.000 99.652 98.449 96.147 92.742
30 100.000 99.652 98.514 96.330 93.069
31 100.000 99.768 98.763 96.699 93.539
32 100.000 99.884 99.012 97.082 94.049
33 100.000 100.000 99.262 97.448 94.508
34 100.000 100.000 99.326 97.635 94.849
35 100.000 100.000 99.372 97.773 95.109
36 100.000 100.000 99.409 97.880 95.308
37 100.000 100.000 99.446 97.986 95.500
38 100.000 100.000 99.483 98.103 95.733
39 100.000 100.000 99.520 98.211 95.931
40 100.000 100.000 99.548 98.304 96.123
41 100.000 100.000 99.575 98.385 96.270
42 100.000 100.000 99.603 98.472 96.442
43 100.000 100.000 99.631 98.551 96.586
44 100.000 100.000 99.658 98.631 96.734
45 100.000 100.000 99.686 98.720 96.911
46 100.000 100.000 99.705 98.781 97.037
47 100.000 100.000 99.723 98.846 97.172
48 100.000 100.000 99.742 98.907 97.299
49 100.000 100.000 99.760 98.970 97.428
50 100.000 100.000 99.778 99.027 97.540
51 100.000 100.000 99.797 99.087 97.660
52 100.000 100.000 99.815 99.147 97.783
53 100.000 100.000 99.825 99.180 97.856
54 100.000 100.000 99.834 99.218 97.944
55 100.000 100.000 99.843 99.247 98.002
56 100.000 100.000 99.852 99.278 98.068
57 100.000 100.000 99.862 99.314 98.148
58 100.000 100.000 99.871 99.343 98.206
59 100.000 100.000 99.880 99.370 98.257
60 100.000 100.000 99.889 99.403 98.330
61 100.000 100.000 99.898 99.442 98.420
62 100.000 100.000 99.908 99.469 98.471
63 100.000 100.000 99.917 99.501 98.540
64 100.000 100.000 99.926 99.528 98.591
65 100.000 100.000 99.935 99.555 98.642
66 100.000 100.000 99.945 99.583 98.697
67 100.000 100.000 99.954 99.619 98.776
68 100.000 100.000 99.963 99.653 98.853
69 100.000 100.000 99.972 99.681 98.908
70 100.000 100.000 99.982 99.708 98.958
71 100.000 100.000 99.991 99.735 99.009
72 100.000 100.000 100.000 99.763 99.060
73 100.000 100.000 100.000 99.771 99.090
74 100.000 100.000 100.000 99.780 99.124
75 100.000 100.000 100.000 99.791 99.161
76 100.000 100.000 100.000 99.797 99.184
77 100.000 100.000 100.000 99.804 99.211
78 100.000 100.000 100.000 99.811 99.237
79 100.000 100.000 100.000 99.818 99.260
80 100.000 100.000 100.000 99.824 99.284
81 100.000 100.000 100.000 99.836 99.328
82 100.000 100.000 100.000 99.844 99.355
83 100.000 100.000 100.000 99.850 99.378
84 100.000 100.000 100.000 99.860 99.415
85 100.000 100.000 100.000 99.869 99.444
86 100.000 100.000 100.000 99.874 99.465
87 100.000 100.000 100.000 99.881 99.491
88 100.000 100.000 100.000 99.885 99.507
89 100.000 100.000 100.000 99.896 99.546
90 100.000 100.000 100.000 99.900 99.561
91 100.000 100.000 100.000 99.903 99.574
92 100.000 100.000 100.000 99.906 99.586
93 100.000 100.000 100.000 99.913 99.609
94 100.000 100.000 100.000 99.917 99.625
95 100.000 100.000 100.000 99.920 99.637
96 100.000 100.000 100.000 99.927 99.664
97 100.000 100.000 100.000 99.932 99.683
98 100.000 100.000 100.000 99.938 99.703
99 100.000 100.000 100.000 99.943 99.723
100 100.000 100.000 100.000 99.950 99.749
101 100.000 100.000 100.000 99.956 99.772
102 100.000 100.000 100.000 99.960 99.788
103 100.000 100.000 100.000 99.966 99.807
104 100.000 100.000 100.000 99.969 99.818
105 100.000 100.000 100.000 99.976 99.845
106 100.000 100.000 100.000 99.982 99.869
107 100.000 100.000 100.000 99.986 99.884
108 100.000 100.000 100.000 99.988 99.889
109 100.000 100.000 100.000 99.994 99.912
110 100.000 100.000 100.000 99.997 99.923
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.503 71.125 60.228 50.409
5 97.674 89.108 78.399 67.511 57.346
6 97.674 90.498 81.169 71.255 61.628
7 97.674 91.425 83.135 74.060 64.987
8 100.000 95.133 87.584 78.814 69.754
9 100.000 95.713 88.877 80.744 72.165
10 100.000 96.524 90.538 83.065 74.914
11 100.000 96.987 91.600 84.684 76.973
12 100.000 97.219 92.227 85.751 78.435
13 100.000 97.683 93.261 87.292 80.357
14 100.000 98.030 94.046 88.486 81.883
15 100.000 98.378 94.831 89.674 83.391
16 100.000 98.378 94.978 90.079 84.096
17 100.000 98.378 95.108 90.439 84.728
18 100.000 98.494 95.468 91.099 85.683
19 100.000 98.610 95.800 91.690 86.525
20 100.000 98.725 96.114 92.236 87.302
21 100.000 98.725 96.215 92.520 87.804
22 100.000 98.957 96.723 93.281 88.774
23 100.000 99.073 97.000 93.736 89.407
24 100.000 99.073 97.083 93.972 89.829
25 100.000 99.189 97.360 94.424 90.452
26 100.000 99.305 97.628 94.853 91.038
27 100.000 99.421 97.895 95.280 91.617
28 100.000 99.421 97.941 95.422 91.891
29 100.000 99.537 98.218 95.866 92.486
30 100.000 99.537 98.265 96.008 92.759
31 100.000 99.537 98.311 96.147 93.024
32 100.000 99.537 98.348 96.264 93.251
33 100.000 99.537 98.403 96.422 93.539
34 100.000 99.537 98.440 96.537 93.761
35 100.000 99.537 98.477 96.649 93.975
36 100.000 99.652 98.726 97.021 94.452
37 100.000 99.652 98.763 97.133 94.663
38 100.000 99.652 98.809 97.264 94.900
39 100.000 99.768 99.058 97.634 95.371
40 100.000 99.768 99.095 97.742 95.572
41 100.000 99.884 99.345 98.110 96.036
42 100.000 100.000 99.594 98.476 96.496
43 100.000 100.000 99.612 98.541 96.633
44 100.000 100.000 99.631 98.604 96.765
45 100.000 100.000 99.658 98.689 96.926
46 100.000 100.000 99.686 98.771 97.081
47 100.000 100.000 99.695 98.812 97.178
48 100.000 100.000 99.714 98.872 97.301
49 100.000 100.000 99.723 98.912 97.395
50 100.000 100.000 99.751 98.993 97.547
51 100.000 100.000 99.760 99.030 97.634
52 100.000 100.000 99.778 99.089 97.749
53 100.000 100.000 99.797 99.147 97.866
54 100.000 100.000 99.815 99.205 97.983
55 100.000 100.000 99.825 99.242 98.066
56 100.000 100.000 99.843 99.298 98.175
57 100.000 100.000 99.852 99.333 98.255
58 100.000 100.000 99.871 99.388 98.356
59 100.000 100.000 99.871 99.399 98.397
60 100.000 100.000 99.880 99.431 98.467
61 100.000 100.000 99.880 99.443 98.509
62 100.000 100.000 99.889 99.474 98.575
63 100.000 100.000 99.898 99.505 98.642
64 100.000 100.000 99.898 99.516 98.681
65 100.000 100.000 99.908 99.546 98.744
66 100.000 100.000 99.908 99.555 98.778
67 100.000 100.000 99.908 99.564 98.808
68 100.000 100.000 99.908 99.572 98.839
69 100.000 100.000 99.908 99.580 98.870
70 100.000 100.000 99.908 99.589 98.900
71 100.000 100.000 99.908 99.596 98.927
72 100.000 100.000 99.917 99.624 98.982
73 100.000 100.000 99.917 99.631 99.008
74 100.000 100.000 99.926 99.659 99.062
75 100.000 100.000 99.935 99.687 99.113
76 100.000 100.000 99.935 99.693 99.137
77 100.000 100.000 99.945 99.720 99.188
78 100.000 100.000 99.954 99.747 99.239
79 100.000 100.000 99.954 99.753 99.262
80 100.000 100.000 99.954 99.759 99.285
81 100.000 100.000 99.954 99.766 99.308
82 100.000 100.000 99.963 99.793 99.358
83 100.000 100.000 99.972 99.820 99.409
84 100.000 100.000 99.982 99.847 99.460
85 100.000 100.000 99.991 99.874 99.511
86 100.000 100.000 100.000 99.901 99.561
87 100.000 100.000 100.000 99.906 99.581
88 100.000 100.000 100.000 99.911 99.600
89 100.000 100.000 100.000 99.917 99.620
90 100.000 100.000 100.000 99.922 99.639
91 100.000 100.000 100.000 99.927 99.659
92 100.000 100.000 100.000 99.932 99.679
93 100.000 100.000 100.000 99.938 99.699
94 100.000 100.000 100.000 99.942 99.714
95 100.000 100.000 100.000 99.946 99.730
96 100.000 100.000 100.000 99.950 99.747
97 100.000 100.000 100.000 99.954 99.762
98 100.000 100.000 100.000 99.958 99.777
99 100.000 100.000 100.000 99.963 99.793
100 100.000 100.000 100.000 99.967 99.808
101 100.000 100.000 100.000 99.971 99.825
102 100.000 100.000 100.000 99.974 99.836
103 100.000 100.000 100.000 99.977 99.848
104 100.000 100.000 100.000 99.980 99.860
105 100.000 100.000 100.000 99.983 99.872
106 100.000 100.000 100.000 99.986 99.884
107 100.000 100.000 100.000 99.990 99.895
108 100.000 100.000 100.000 99.993 99.907
109 100.000 100.000 100.000 99.996 99.918
110 100.000 100.000 100.000 99.999 99.930
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 79.070 62.109 48.777 38.407 30.360
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.374 68.014 57.508 48.199
5 93.023 83.314 72.925 62.880 53.659
6 95.349 87.486 78.252 68.754 59.628
7 97.674 91.194 82.756 73.646 64.612
8 97.674 91.889 84.252 75.820 67.271
9 100.000 95.597 88.655 80.467 71.882
10 100.000 96.524 90.529 83.049 74.900
11 100.000 96.987 91.554 84.578 76.819
12 100.000 97.219 92.209 85.710 78.380
13 100.000 97.683 93.215 87.193 80.227
14 100.000 98.030 94.018 88.434 81.828
15 100.000 98.262 94.581 89.341 83.042
16 100.000 98.262 94.729 89.742 83.734
17 100.000 98.262 94.877 90.140 84.416
18 100.000 98.262 94.969 90.404 84.895
19 100.000 98.262 95.052 90.645 85.335
20 100.000 98.262 95.154 90.926 85.830
21 100.000 98.494 95.717 91.823 87.010
22 100.000 98.725 96.252 92.653 88.089
23 100.000 98.841 96.520 93.084 88.683
24 100.000 98.957 96.815 93.584 89.384
25 100.000 99.073 97.083 94.015 89.978
26 100.000 99.073 97.175 94.272 90.429
27 100.000 99.073 97.231 94.436 90.737
28 100.000 99.073 97.295 94.622 91.074
29 100.000 99.073 97.341 94.763 91.346
30 100.000 99.189 97.600 95.163 91.877
31 100.000 99.189 97.665 95.350 92.216
32 100.000 99.305 97.923 95.750 92.747
33 100.000 99.305 97.969 95.889 93.011
34 100.000 99.305 98.006 96.003 93.230
35 100.000 99.305 98.043 96.112 93.436
36 100.000 99.305 98.080 96.222 93.645
37 100.000 99.305 98.089 96.265 93.749
38 100.000 99.305 98.117 96.354 93.925
39 100.000 99.421 98.366 96.726 94.405
40 100.000 99.537 98.615 97.098 94.883
41 100.000 99.652 98.865 97.469 95.357
42 100.000 99.652 98.883 97.533 95.490
43 100.000 99.652 98.902 97.596 95.622
44 100.000 99.652 98.911 97.636 95.716
45 100.000 99.652 98.920 97.677 95.815
46 100.000 99.652 98.948 97.760 95.971
47 100.000 99.652 98.966 97.820 96.093
48 100.000 99.652 98.994 97.900 96.242
49 100.000 99.652 99.012 97.958 96.355
50 100.000 99.652 99.012 97.968 96.393
51 100.000 99.652 99.012 97.976 96.425
52 100.000 99.652 99.022 98.011 96.501
53 100.000 99.652 99.040 98.067 96.611
54 100.000 99.768 99.289 98.434 97.070
55 100.000 99.884 99.538 98.799 97.527
56 100.000 99.884 99.548 98.836 97.609
57 100.000 99.884 99.557 98.871 97.689
58 100.000 100.000 99.806 99.237 98.145
59 100.000 100.000 99.825 99.292 98.251
60 100.000 100.000 99.834 99.324 98.320
61 100.000 100.000 99.843 99.355 98.387
62 100.000 100.000 99.852 99.385 98.450
63 100.000 100.000 99.862 99.416 98.513
64 100.000 100.000 99.862 99.425 98.547
65 100.000 100.000 99.862 99.432 98.574
66 100.000 100.000 99.862 99.442 98.607
67 100.000 100.000 99.862 99.448 98.631
68 100.000 100.000 99.880 99.502 98.733
69 100.000 100.000 99.889 99.534 98.802
70 100.000 100.000 99.889 99.542 98.829
71 100.000 100.000 99.889 99.549 98.855
72 100.000 100.000 99.889 99.554 98.875
73 100.000 100.000 99.889 99.560 98.898
74 100.000 100.000 99.889 99.565 98.915
75 100.000 100.000 99.889 99.571 98.939
76 100.000 100.000 99.908 99.625 99.040
77 100.000 100.000 99.908 99.631 99.063
78 100.000 100.000 99.908 99.639 99.089
79 100.000 100.000 99.908 99.644 99.108
80 100.000 100.000 99.908 99.649 99.128
81 100.000 100.000 99.908 99.654 99.147
82 100.000 100.000 99.908 99.660 99.170
83 100.000 100.000 99.908 99.666 99.189
84 100.000 100.000 99.908 99.671 99.209
85 100.000 100.000 99.908 99.675 99.225
86 100.000 100.000 99.908 99.679 99.240
87 100.000 100.000 99.908 99.683 99.256
88 100.000 100.000 99.908 99.688 99.272
89 100.000 100.000 99.908 99.692 99.287
90 100.000 100.000 99.908 99.696 99.302
91 100.000 100.000 99.908 99.700 99.318
92 100.000 100.000 99.908 99.703 99.330
93 100.000 100.000 99.908 99.706 99.342
94 100.000 100.000 99.917 99.733 99.393
95 100.000 100.000 99.917 99.736 99.404
96 100.000 100.000 99.926 99.764 99.455
97 100.000 100.000 99.935 99.791 99.506
98 100.000 100.000 99.945 99.818 99.557
99 100.000 100.000 99.954 99.845 99.608
100 100.000 100.000 99.954 99.848 99.619
101 100.000 100.000 99.963 99.875 99.670
102 100.000 100.000 99.972 99.902 99.721
103 100.000 100.000 99.972 99.905 99.732
104 100.000 100.000 99.972 99.908 99.744
105 100.000 100.000 99.982 99.935 99.795
106 100.000 100.000 99.991 99.963 99.845
107 100.000 100.000 100.000 99.990 99.896
108 100.000 100.000 100.000 99.993 99.907
109 100.000 100.000 100.000 99.996 99.919
110 100.000 100.000 100.000 99.999 99.930
111 100.000 100.000 100.000 100.000 99.935

flex

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 90.698 80.185 69.371 59.041 49.654
5 95.349 86.790 76.645 66.321 56.585
6 100.000 93.163 83.486 73.051 62.925
7 100.000 94.322 85.876 76.378 66.822
8 100.000 95.365 87.935 79.158 70.018
9 100.000 96.176 89.597 81.480 72.770
10 100.000 96.871 90.954 83.288 74.819
11 100.000 97.335 91.997 84.858 76.797
12 100.000 97.683 92.754 85.966 78.159
13 100.000 97.914 93.298 86.817 79.267
14 100.000 98.146 93.908 87.839 80.660
15 100.000 98.378 94.498 88.802 81.938
16 100.000 98.610 95.080 89.746 83.196
17 100.000 98.841 95.661 90.688 84.445
18 100.000 98.957 95.920 91.085 84.964
19 100.000 99.073 96.243 91.648 85.757
20 100.000 99.189 96.612 92.332 86.750
21 100.000 99.305 96.889 92.779 87.355
22 100.000 99.421 97.166 93.223 87.950
23 100.000 99.537 97.415 93.588 88.405
24 100.000 99.652 97.665 93.963 88.892
25 100.000 99.768 97.960 94.456 89.569
26 100.000 99.884 98.218 94.854 90.091
27 100.000 100.000 98.468 95.225 90.564
28 100.000 100.000 98.495 95.305 90.711
29 100.000 100.000 98.615 95.634 91.284
30 100.000 100.000 98.745 95.990 91.906
31 100.000 100.000 98.818 96.202 92.289
32 100.000 100.000 98.883 96.382 92.608
33 100.000 100.000 98.929 96.520 92.869
34 100.000 100.000 98.975 96.655 93.116
35 100.000 100.000 99.012 96.766 93.327
36 100.000 100.000 99.058 96.899 93.571
37 100.000 100.000 99.105 97.033 93.815
38 100.000 100.000 99.142 97.147 94.037
39 100.000 100.000 99.151 97.180 94.108
40 100.000 100.000 99.188 97.286 94.302
41 100.000 100.000 99.262 97.501 94.699
42 100.000 100.000 99.280 97.569 94.845
43 100.000 100.000 99.298 97.633 94.978
44 100.000 100.000 99.326 97.718 95.144
45 100.000 100.000 99.326 97.733 95.195
46 100.000 100.000 99.354 97.816 95.355
47 100.000 100.000 99.382 97.898 95.511
48 100.000 100.000 99.428 98.026 95.737
49 100.000 100.000 99.465 98.137 95.946
50 100.000 100.000 99.483 98.194 96.058
51 100.000 100.000 99.511 98.276 96.213
52 100.000 100.000 99.511 98.285 96.243
53 100.000 100.000 99.520 98.325 96.340
54 100.000 100.000 99.529 98.354 96.399
55 100.000 100.000 99.548 98.409 96.500
56 100.000 100.000 99.594 98.543 96.747
57 100.000 100.000 99.594 98.548 96.768
58 100.000 100.000 99.603 98.577 96.827
59 100.000 100.000 99.631 98.664 96.996
60 100.000 100.000 99.658 98.748 97.158
61 100.000 100.000 99.686 98.828 97.305
62 100.000 100.000 99.695 98.864 97.386
63 100.000 100.000 99.695 98.868 97.404
64 100.000 100.000 99.705 98.898 97.467
65 100.000 100.000 99.714 98.926 97.521
66 100.000 100.000 99.714 98.929 97.533
67 100.000 100.000 99.732 98.990 97.656
68 100.000 100.000 99.751 99.053 97.789
69 100.000 100.000 99.760 99.086 97.858
70 100.000 100.000 99.760 99.089 97.869
71 100.000 100.000 99.760 99.098 97.903
72 100.000 100.000 99.778 99.153 98.008
73 100.000 100.000 99.788 99.180 98.059
74 100.000 100.000 99.797 99.213 98.129
75 100.000 100.000 99.806 99.242 98.188
76 100.000 100.000 99.825 99.295 98.286
77 100.000 100.000 99.834 99.322 98.336
78 100.000 100.000 99.843 99.350 98.391
79 100.000 100.000 99.843 99.354 98.407
80 100.000 100.000 99.852 99.381 98.457
81 100.000 100.000 99.862 99.410 98.516
82 100.000 100.000 99.862 99.413 98.524
83 100.000 100.000 99.871 99.448 98.604
84 100.000 100.000 99.871 99.454 98.627
85 100.000 100.000 99.880 99.481 98.677
86 100.000 100.000 99.880 99.483 98.686
87 100.000 100.000 99.880 99.488 98.702
88 100.000 100.000 99.880 99.488 98.702
89 100.000 100.000 99.889 99.518 98.764
90 100.000 100.000 99.908 99.579 98.891
91 100.000 100.000 99.908 99.583 98.907
92 100.000 100.000 99.908 99.587 98.918
93 100.000 100.000 99.908 99.591 98.934
94 100.000 100.000 99.917 99.622 99.000
95 100.000 100.000 99.935 99.677 99.106
96 100.000 100.000 99.935 99.677 99.106
97 100.000 100.000 99.935 99.688 99.143
98 100.000 100.000 99.935 99.691 99.155
99 100.000 100.000 99.945 99.718 99.206
100 100.000 100.000 99.945 99.721 99.218
101 100.000 100.000 99.945 99.724 99.229
102 100.000 100.000 99.945 99.724 99.230
103 100.000 100.000 99.945 99.729 99.249
104 100.000 100.000 99.945 99.733 99.265
105 100.000 100.000 99.954 99.761 99.321
106 100.000 100.000 99.954 99.766 99.336
107 100.000 100.000 99.954 99.768 99.344
108 100.000 100.000 99.954 99.769 99.348
109 100.000 100.000 99.954 99.773 99.364
110 100.000 100.000 99.963 99.801 99.420
111 100.000 100.000 99.963 99.805 99.435
112 100.000 100.000 99.963 99.806 99.440
113 100.000 100.000 99.963 99.809 99.452
114 100.000 100.000 99.963 99.813 99.464
115 100.000 100.000 99.963 99.815 99.473
116 100.000 100.000 99.963 99.819 99.488
117 100.000 100.000 99.963 99.821 99.496
118 100.000 100.000 99.963 99.822 99.500
119 100.000 100.000 99.963 99.826 99.516
120 100.000 100.000 99.963 99.828 99.525
121 100.000 100.000 99.963 99.829 99.530
122 100.000 100.000 99.963 99.833 99.545
123 100.000 100.000 99.963 99.834 99.550
124 100.000 100.000 99.972 99.861 99.601
125 100.000 100.000 99.972 99.861 99.603
126 100.000 100.000 99.972 99.861 99.603
127 100.000 100.000 99.972 99.863 99.608
128 100.000 100.000 99.982 99.891 99.662
129 100.000 100.000 99.982 99.894 99.675
130 100.000 100.000 99.982 99.897 99.687
131 100.000 100.000 99.982 99.899 99.695
132 100.000 100.000 99.982 99.899 99.696
133 100.000 100.000 99.982 99.903 99.712
134 100.000 100.000 99.982 99.905 99.720
135 100.000 100.000 99.982 99.907 99.728
136 100.000 100.000 99.982 99.908 99.733
137 100.000 100.000 99.982 99.911 99.745
138 100.000 100.000 99.982 99.914 99.753
139 100.000 100.000 99.982 99.916 99.761
140 100.000 100.000 99.982 99.919 99.772
141 100.000 100.000 99.982 99.919 99.773
142 100.000 100.000 99.982 99.920 99.777
143 100.000 100.000 99.982 99.922 99.785
144 100.000 100.000 99.982 99.924 99.794
145 100.000 100.000 99.982 99.924 99.796
146 100.000 100.000 99.982 99.926 99.804
147 100.000 100.000 99.991 99.953 99.854
148 100.000 100.000 99.991 99.953 99.855
149 100.000 100.000 99.991 99.955 99.863
150 100.000 100.000 99.991 99.956 99.867
151 100.000 100.000 99.991 99.956 99.867
152 100.000 100.000 99.991 99.958 99.875
153 100.000 100.000 99.991 99.959 99.880
154 100.000 100.000 99.991 99.959 99.881
155 100.000 100.000 99.991 99.961 99.889
156 100.000 100.000 99.991 99.964 99.896
157 100.000 100.000 99.991 99.964 99.898
158 100.000 100.000 99.991 99.965 99.902
159 100.000 100.000 99.991 99.965 99.903
160 100.000 100.000 99.991 99.965 99.904
161 100.000 100.000 99.991 99.965 99.905
162 100.000 100.000 99.991 99.967 99.913
163 100.000 100.000 99.991 99.968 99.918
164 100.000 100.000 99.991 99.969 99.922
165 100.000 100.000 99.991 99.969 99.923
166 100.000 100.000 99.991 99.969 99.924
167 100.000 100.000 100.000 99.996 99.975
168 100.000 100.000 100.000 99.997 99.979
169 100.000 100.000 100.000 99.998 99.984
170 100.000 100.000 100.000 99.998 99.984
171 100.000 100.000 100.000 99.998 99.985
172 100.000 100.000 100.000 99.998 99.986
173 100.000 100.000 100.000 99.998 99.987
174 100.000 100.000 100.000 99.998 99.988
175 100.000 100.000 100.000 99.999 99.992
176 100.000 100.000 100.000 99.999 99.993
177 100.000 100.000 100.000 99.999 99.994
178 100.000 100.000 100.000 100.000 99.998
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 83.721 67.323 53.245 41.856 32.878
3 90.698 77.752 64.802 53.194 43.303
4 93.023 82.619 71.291 60.381 50.518
5 95.349 86.559 76.230 65.817 56.069
6 97.674 90.382 80.947 70.967 61.313
7 97.674 91.425 83.153 74.099 65.039
8 100.000 95.249 87.778 79.027 69.942
9 100.000 96.176 89.661 81.632 72.997
10 100.000 96.871 91.138 83.755 75.571
11 100.000 97.219 92.015 85.177 77.453
12 100.000 97.567 92.837 86.465 79.132
13 100.000 97.914 93.658 87.745 80.785
14 100.000 98.146 94.258 88.733 82.106
15 100.000 98.378 94.849 89.712 83.437
16 100.000 98.378 95.034 90.202 84.266
17 100.000 98.610 95.588 91.081 85.426
18 100.000 98.725 95.938 91.726 86.367
19 100.000 98.841 96.280 92.330 87.214
20 100.000 98.957 96.612 92.923 88.063
21 100.000 99.189 97.111 93.656 88.981
22 100.000 99.305 97.406 94.158 89.689
23 100.000 99.305 97.498 94.418 90.154
24 100.000 99.305 97.581 94.650 90.562
25 100.000 99.421 97.840 95.051 91.095
26 100.000 99.537 98.098 95.446 91.611
27 100.000 99.537 98.172 95.659 91.996
28 100.000 99.652 98.421 96.026 92.456
29 100.000 99.768 98.671 96.391 92.911
30 100.000 99.884 98.920 96.768 93.404
31 100.000 99.884 98.985 96.955 93.743
32 100.000 100.000 99.234 97.322 94.206
33 100.000 100.000 99.289 97.480 94.488
34 100.000 100.000 99.335 97.616 94.742
35 100.000 100.000 99.382 97.758 95.013
36 100.000 100.000 99.428 97.900 95.288
37 100.000 100.000 99.465 98.020 95.527
38 100.000 100.000 99.502 98.136 95.753
39 100.000 100.000 99.538 98.240 95.940
40 100.000 100.000 99.575 98.355 96.167
41 100.000 100.000 99.612 98.460 96.354
42 100.000 100.000 99.649 98.570 96.563
43 100.000 100.000 99.686 98.684 96.782
44 100.000 100.000 99.723 98.791 96.980
45 100.000 100.000 99.751 98.878 97.151
46 100.000 100.000 99.778 98.961 97.307
47 100.000 100.000 99.806 99.051 97.489
48 100.000 100.000 99.834 99.139 97.663
49 100.000 100.000 99.852 99.196 97.775
50 100.000 100.000 99.871 99.254 97.892
51 100.000 100.000 99.880 99.281 97.943
52 100.000 100.000 99.889 99.316 98.019
53 100.000 100.000 99.898 99.351 98.100
54 100.000 100.000 99.908 99.382 98.167
55 100.000 100.000 99.917 99.409 98.218
56 100.000 100.000 99.926 99.438 98.273
57 100.000 100.000 99.935 99.475 98.360
58 100.000 100.000 99.945 99.505 98.421
59 100.000 100.000 99.954 99.532 98.473
60 100.000 100.000 99.963 99.560 98.527
61 100.000 100.000 99.972 99.588 98.578
62 100.000 100.000 99.982 99.616 98.632
63 100.000 100.000 99.991 99.643 98.683
64 100.000 100.000 100.000 99.671 98.737
65 100.000 100.000 100.000 99.675 98.752
66 100.000 100.000 100.000 99.685 98.789
67 100.000 100.000 100.000 99.694 98.819
68 100.000 100.000 100.000 99.699 98.839
69 100.000 100.000 100.000 99.708 98.873
70 100.000 100.000 100.000 99.717 98.904
71 100.000 100.000 100.000 99.726 98.938
72 100.000 100.000 100.000 99.736 98.975
73 100.000 100.000 100.000 99.743 98.997
74 100.000 100.000 100.000 99.748 99.017
75 100.000 100.000 100.000 99.753 99.036
76 100.000 100.000 100.000 99.759 99.059
77 100.000 100.000 100.000 99.761 99.067
78 100.000 100.000 100.000 99.767 99.087
79 100.000 100.000 100.000 99.772 99.106
80 100.000 100.000 100.000 99.779 99.132
81 100.000 100.000 100.000 99.788 99.162
82 100.000 100.000 100.000 99.793 99.182
83 100.000 100.000 100.000 99.799 99.205
84 100.000 100.000 100.000 99.803 99.222
85 100.000 100.000 100.000 99.806 99.233
86 100.000 100.000 100.000 99.816 99.267
87 100.000 100.000 100.000 99.823 99.293
88 100.000 100.000 100.000 99.828 99.313
89 100.000 100.000 100.000 99.832 99.329
90 100.000 100.000 100.000 99.838 99.348
91 100.000 100.000 100.000 99.841 99.360
92 100.000 100.000 100.000 99.846 99.380
93 100.000 100.000 100.000 99.851 99.399
94 100.000 100.000 100.000 99.855 99.415
95 100.000 100.000 100.000 99.856 99.419
96 100.000 100.000 100.000 99.860 99.435
97 100.000 100.000 100.000 99.865 99.450
98 100.000 100.000 100.000 99.869 99.466
99 100.000 100.000 100.000 99.871 99.474
100 100.000 100.000 100.000 99.877 99.497
101 100.000 100.000 100.000 99.880 99.508
102 100.000 100.000 100.000 99.885 99.528
103 100.000 100.000 100.000 99.890 99.544
104 100.000 100.000 100.000 99.894 99.559
105 100.000 100.000 100.000 99.897 99.571
106 100.000 100.000 100.000 99.900 99.583
107 100.000 100.000 100.000 99.901 99.587
108 100.000 100.000 100.000 99.902 99.591
109 100.000 100.000 100.000 99.905 99.604
110 100.000 100.000 100.000 99.908 99.616
111 100.000 100.000 100.000 99.913 99.632
112 100.000 100.000 100.000 99.915 99.641
113 100.000 100.000 100.000 99.916 99.646
114 100.000 100.000 100.000 99.916 99.648
115 100.000 100.000 100.000 99.919 99.660
116 100.000 100.000 100.000 99.922 99.671
117 100.000 100.000 100.000 99.924 99.679
118 100.000 100.000 100.000 99.927 99.691
119 100.000 100.000 100.000 99.928 99.696
120 100.000 100.000 100.000 99.934 99.719
121 100.000 100.000 100.000 99.938 99.731
122 100.000 100.000 100.000 99.940 99.739
123 100.000 100.000 100.000 99.941 99.744
124 100.000 100.000 100.000 99.944 99.756
125 100.000 100.000 100.000 99.944 99.757
126 100.000 100.000 100.000 99.946 99.765
127 100.000 100.000 100.000 99.947 99.769
128 100.000 100.000 100.000 99.947 99.770
129 100.000 100.000 100.000 99.948 99.775
130 100.000 100.000 100.000 99.948 99.775
131 100.000 100.000 100.000 99.950 99.783
132 100.000 100.000 100.000 99.951 99.788
133 100.000 100.000 100.000 99.955 99.803
134 100.000 100.000 100.000 99.959 99.819
135 100.000 100.000 100.000 99.960 99.824
136 100.000 100.000 100.000 99.963 99.832
137 100.000 100.000 100.000 99.963 99.833
138 100.000 100.000 100.000 99.964 99.837
139 100.000 100.000 100.000 99.966 99.846
140 100.000 100.000 100.000 99.967 99.851
141 100.000 100.000 100.000 99.969 99.859
142 100.000 100.000 100.000 99.970 99.864
143 100.000 100.000 100.000 99.970 99.864
144 100.000 100.000 100.000 99.971 99.869
145 100.000 100.000 100.000 99.973 99.877
146 100.000 100.000 100.000 99.974 99.881
147 100.000 100.000 100.000 99.977 99.893
148 100.000 100.000 100.000 99.980 99.904
149 100.000 100.000 100.000 99.982 99.912
150 100.000 100.000 100.000 99.982 99.913
151 100.000 100.000 100.000 99.983 99.918
152 100.000 100.000 100.000 99.985 99.926
153 100.000 100.000 100.000 99.988 99.934
154 100.000 100.000 100.000 99.991 99.945
155 100.000 100.000 100.000 99.991 99.946
156 100.000 100.000 100.000 99.992 99.952
157 100.000 100.000 100.000 99.994 99.959
158 100.000 100.000 100.000 99.996 99.967
159 100.000 100.000 100.000 99.997 99.972
160 100.000 100.000 100.000 99.997 99.972
161 100.000 100.000 100.000 99.997 99.973
162 100.000 100.000 100.000 99.997 99.974
163 100.000 100.000 100.000 99.997 99.975
164 100.000 100.000 100.000 99.997 99.977
165 100.000 100.000 100.000 99.997 99.978
166 100.000 100.000 100.000 99.997 99.979
167 100.000 100.000 100.000 99.997 99.980
168 100.000 100.000 100.000 99.998 99.985
169 100.000 100.000 100.000 99.998 99.986
170 100.000 100.000 100.000 99.999 99.990
171 100.000 100.000 100.000 99.999 99.991
172 100.000 100.000 100.000 99.999 99.992
173 100.000 100.000 100.000 99.999 99.992
174 100.000 100.000 100.000 99.999 99.993
175 100.000 100.000 100.000 99.999 99.993
176 100.000 100.000 100.000 99.999 99.994
177 100.000 100.000 100.000 100.000 99.998
178 100.000 100.000 100.000 100.000 99.999
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 90.698 79.722 68.596 58.165 48.823
5 95.349 86.327 75.870 65.445 55.754
6 97.674 90.267 80.809 70.881 61.305
7 97.674 90.846 82.110 72.833 63.749
8 100.000 94.322 86.135 77.062 67.971
9 100.000 95.017 87.630 79.227 70.602
10 100.000 96.060 89.689 82.005 73.792
11 100.000 96.640 90.907 83.753 75.921
12 100.000 97.103 91.951 85.325 77.902
13 100.000 97.567 92.966 86.827 79.771
14 100.000 97.914 93.788 88.109 81.431
15 100.000 98.030 94.175 88.844 82.513
16 100.000 98.146 94.517 89.467 83.422
17 100.000 98.262 94.849 90.067 84.294
18 100.000 98.494 95.384 90.898 85.376
19 100.000 98.725 95.920 91.727 86.451
20 100.000 98.725 96.012 91.993 86.934
21 100.000 98.957 96.529 92.778 87.944
22 100.000 99.073 96.852 93.343 88.740
23 100.000 99.189 97.157 93.864 89.474
24 100.000 99.189 97.240 94.099 89.892
25 100.000 99.305 97.526 94.572 90.544
26 100.000 99.305 97.609 94.805 90.960
27 100.000 99.305 97.692 95.036 91.368
28 100.000 99.305 97.757 95.226 91.718
29 100.000 99.305 97.821 95.413 92.062
30 100.000 99.305 97.886 95.599 92.399
31 100.000 99.305 97.932 95.737 92.661
32 100.000 99.305 97.988 95.894 92.945
33 100.000 99.305 98.034 96.031 93.199
34 100.000 99.421 98.292 96.427 93.715
35 100.000 99.421 98.320 96.519 93.904
36 100.000 99.421 98.348 96.610 94.087
37 100.000 99.537 98.597 96.982 94.564
38 100.000 99.537 98.615 97.050 94.713
39 100.000 99.537 98.652 97.161 94.923
40 100.000 99.652 98.902 97.531 95.394
41 100.000 99.652 98.938 97.639 95.596
42 100.000 99.768 99.188 98.008 96.062
43 100.000 99.884 99.437 98.376 96.529
44 100.000 99.884 99.474 98.482 96.724
45 100.000 99.884 99.502 98.568 96.891
46 100.000 100.000 99.751 98.933 97.347
47 100.000 100.000 99.769 98.997 97.480
48 100.000 100.000 99.806 99.101 97.667
49 100.000 100.000 99.825 99.163 97.794
50 100.000 100.000 99.843 99.224 97.920
51 100.000 100.000 99.852 99.262 98.007
52 100.000 100.000 99.871 99.319 98.120
53 100.000 100.000 99.880 99.354 98.200
54 100.000 100.000 99.898 99.408 98.302
55 100.000 100.000 99.898 99.421 98.348
56 100.000 100.000 99.898 99.432 98.388
57 100.000 100.000 99.908 99.465 98.457
58 100.000 100.000 99.908 99.476 98.497
59 100.000 100.000 99.908 99.487 98.533
60 100.000 100.000 99.908 99.496 98.567
61 100.000 100.000 99.908 99.504 98.597
62 100.000 100.000 99.908 99.513 98.627
63 100.000 100.000 99.908 99.520 98.654
64 100.000 100.000 99.908 99.527 98.680
65 100.000 100.000 99.908 99.534 98.707
66 100.000 100.000 99.908 99.541 98.730
67 100.000 100.000 99.917 99.568 98.780
68 100.000 100.000 99.917 99.574 98.803
69 100.000 100.000 99.926 99.601 98.854
70 100.000 100.000 99.926 99.607 98.877
71 100.000 100.000 99.935 99.634 98.928
72 100.000 100.000 99.945 99.662 98.979
73 100.000 100.000 99.945 99.668 99.002
74 100.000 100.000 99.954 99.695 99.053
75 100.000 100.000 99.954 99.701 99.076
76 100.000 100.000 99.963 99.728 99.127
77 100.000 100.000 99.972 99.755 99.177
78 100.000 100.000 99.982 99.782 99.229
79 100.000 100.000 99.991 99.809 99.280
80 100.000 100.000 99.991 99.816 99.303
81 100.000 100.000 100.000 99.843 99.354
82 100.000 100.000 100.000 99.848 99.374
83 100.000 100.000 100.000 99.853 99.393
84 100.000 100.000 100.000 99.858 99.413
85 100.000 100.000 100.000 99.864 99.432
86 100.000 100.000 100.000 99.869 99.451
87 100.000 100.000 100.000 99.873 99.467
88 100.000 100.000 100.000 99.877 99.482
89 100.000 100.000 100.000 99.881 99.498
90 100.000 100.000 100.000 99.885 99.514
91 100.000 100.000 100.000 99.890 99.530
92 100.000 100.000 100.000 99.894 99.547
93 100.000 100.000 100.000 99.898 99.562
94 100.000 100.000 100.000 99.902 99.578
95 100.000 100.000 100.000 99.906 99.593
96 100.000 100.000 100.000 99.910 99.609
97 100.000 100.000 100.000 99.915 99.624
98 100.000 100.000 100.000 99.918 99.636
99 100.000 100.000 100.000 99.921 99.648
100 100.000 100.000 100.000 99.924 99.660
101 100.000 100.000 100.000 99.927 99.671
102 100.000 100.000 100.000 99.930 99.684
103 100.000 100.000 100.000 99.933 99.696
104 100.000 100.000 100.000 99.936 99.708
105 100.000 100.000 100.000 99.940 99.721
106 100.000 100.000 100.000 99.943 99.733
107 100.000 100.000 100.000 99.945 99.742
108 100.000 100.000 100.000 99.947 99.750
109 100.000 100.000 100.000 99.949 99.758
110 100.000 100.000 100.000 99.951 99.766
111 100.000 100.000 100.000 99.953 99.774
112 100.000 100.000 100.000 99.955 99.782
113 100.000 100.000 100.000 99.957 99.790
114 100.000 100.000 100.000 99.959 99.799
115 100.000 100.000 100.000 99.961 99.807
116 100.000 100.000 100.000 99.964 99.815
117 100.000 100.000 100.000 99.966 99.823
118 100.000 100.000 100.000 99.968 99.831
119 100.000 100.000 100.000 99.970 99.840
120 100.000 100.000 100.000 99.972 99.848
121 100.000 100.000 100.000 99.974 99.857
122 100.000 100.000 100.000 99.976 99.865
123 100.000 100.000 100.000 99.978 99.873
124 100.000 100.000 100.000 99.980 99.881
125 100.000 100.000 100.000 99.982 99.889
126 100.000 100.000 100.000 99.984 99.897
127 100.000 100.000 100.000 99.986 99.905
128 100.000 100.000 100.000 99.989 99.913
129 100.000 100.000 100.000 99.991 99.921
130 100.000 100.000 100.000 99.993 99.929
131 100.000 100.000 100.000 99.994 99.934
132 100.000 100.000 100.000 99.995 99.939
133 100.000 100.000 100.000 99.996 99.944
134 100.000 100.000 100.000 99.997 99.949
135 100.000 100.000 100.000 99.998 99.954
136 100.000 100.000 100.000 99.999 99.959
137 100.000 100.000 100.000 100.000 99.964
138 100.000 100.000 100.000 100.000 99.964
139 100.000 100.000 100.000 100.000 99.965
140 100.000 100.000 100.000 100.000 99.966
141 100.000 100.000 100.000 100.000 99.967
142 100.000 100.000 100.000 100.000 99.967
143 100.000 100.000 100.000 100.000 99.968
144 100.000 100.000 100.000 100.000 99.969
145 100.000 100.000 100.000 100.000 99.970
146 100.000 100.000 100.000 100.000 99.971
147 100.000 100.000 100.000 100.000 99.972
148 100.000 100.000 100.000 100.000 99.973
149 100.000 100.000 100.000 100.000 99.974
150 100.000 100.000 100.000 100.000 99.974
151 100.000 100.000 100.000 100.000 99.975
152 100.000 100.000 100.000 100.000 99.977
153 100.000 100.000 100.000 100.000 99.977
154 100.000 100.000 100.000 100.000 99.978
155 100.000 100.000 100.000 100.000 99.979
156 100.000 100.000 100.000 100.000 99.979
157 100.000 100.000 100.000 100.000 99.981
158 100.000 100.000 100.000 100.000 99.981
159 100.000 100.000 100.000 100.000 99.982
160 100.000 100.000 100.000 100.000 99.983
161 100.000 100.000 100.000 100.000 99.984
162 100.000 100.000 100.000 100.000 99.984
163 100.000 100.000 100.000 100.000 99.986
164 100.000 100.000 100.000 100.000 99.986
165 100.000 100.000 100.000 100.000 99.988
166 100.000 100.000 100.000 100.000 99.988
167 100.000 100.000 100.000 100.000 99.990
168 100.000 100.000 100.000 100.000 99.990
169 100.000 100.000 100.000 100.000 99.991
170 100.000 100.000 100.000 100.000 99.992
171 100.000 100.000 100.000 100.000 99.993
172 100.000 100.000 100.000 100.000 99.994
173 100.000 100.000 100.000 100.000 99.994
174 100.000 100.000 100.000 100.000 99.995
175 100.000 100.000 100.000 100.000 99.995
176 100.000 100.000 100.000 100.000 99.996
177 100.000 100.000 100.000 100.000 99.997
178 100.000 100.000 100.000 100.000 99.998
179 100.000 100.000 100.000 100.000 99.999
180 100.000 100.000 100.000 100.000 100.000

flex

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 67.442 47.045 33.730 24.737 18.489
2 81.395 64.774 51.112 40.252 31.739
3 88.372 74.855 62.088 50.933 41.540
4 93.023 82.503 71.125 60.228 50.409
5 95.349 86.327 75.852 65.403 55.691
6 95.349 87.601 78.445 68.973 59.835
7 95.349 88.528 80.384 71.714 63.102
8 95.349 89.108 81.723 73.754 65.676
9 97.674 92.932 86.357 78.710 70.634
10 97.674 93.163 87.012 79.849 72.214
11 97.674 93.859 88.461 81.905 74.684
12 97.674 94.670 90.067 84.089 77.221
13 100.000 97.914 93.640 87.720 80.778
14 100.000 98.030 94.018 88.436 81.834
15 100.000 98.262 94.628 89.451 83.208
16 100.000 98.378 95.006 90.161 84.244
17 100.000 98.494 95.357 90.801 85.172
18 100.000 98.494 95.486 91.157 85.795
19 100.000 98.494 95.597 91.468 86.346
20 100.000 98.610 95.901 91.991 87.084
21 100.000 98.610 95.994 92.247 87.536
22 100.000 98.841 96.492 92.989 88.483
23 100.000 98.841 96.594 93.272 88.980
24 100.000 99.073 97.101 94.033 89.950
25 100.000 99.073 97.185 94.264 90.360
26 100.000 99.073 97.240 94.434 90.684
27 100.000 99.073 97.286 94.576 90.956
28 100.000 99.073 97.323 94.691 91.183
29 100.000 99.073 97.406 94.925 91.597
30 100.000 99.073 97.434 95.017 91.787
31 100.000 99.073 97.461 95.109 91.974
32 100.000 99.189 97.729 95.533 92.543
33 100.000 99.189 97.775 95.672 92.807
34 100.000 99.189 97.794 95.736 92.942
35 100.000 99.305 98.043 96.111 93.430
36 100.000 99.305 98.052 96.151 93.525
37 100.000 99.305 98.098 96.284 93.769
38 100.000 99.305 98.145 96.416 94.010
39 100.000 99.305 98.154 96.454 94.098
40 100.000 99.305 98.181 96.540 94.267
41 100.000 99.305 98.218 96.646 94.462
42 100.000 99.421 98.477 97.038 94.965
43 100.000 99.421 98.514 97.144 95.161
44 100.000 99.421 98.532 97.205 95.284
45 100.000 99.421 98.551 97.264 95.403
46 100.000 99.421 98.560 97.299 95.484
47 100.000 99.537 98.809 97.666 95.944
48 100.000 99.537 98.818 97.703 96.031
49 100.000 99.652 99.068 98.071 96.494
50 100.000 99.652 99.077 98.105 96.571
51 100.000 99.652 99.086 98.141 96.650
52 100.000 99.768 99.335 98.507 97.110
53 100.000 99.884 99.585 98.874 97.569
54 100.000 99.884 99.603 98.931 97.681
55 100.000 99.884 99.612 98.963 97.748
56 100.000 99.884 99.612 98.972 97.783
57 100.000 99.884 99.622 99.006 97.858
58 100.000 100.000 99.871 99.372 98.314
59 100.000 100.000 99.889 99.429 98.426
60 100.000 100.000 99.889 99.439 98.460
61 100.000 100.000 99.889 99.450 98.500
62 100.000 100.000 99.889 99.457 98.527
63 100.000 100.000 99.889 99.466 98.557
64 100.000 100.000 99.889 99.473 98.584
65 100.000 100.000 99.889 99.480 98.611
66 100.000 100.000 99.908 99.533 98.709
67 100.000 100.000 99.908 99.542 98.739
68 100.000 100.000 99.908 99.548 98.762
69 100.000 100.000 99.908 99.556 98.792
70 100.000 100.000 99.908 99.563 98.815
71 100.000 100.000 99.908 99.569 98.838
72 100.000 100.000 99.908 99.573 98.855
73 100.000 100.000 99.908 99.578 98.875
74 100.000 100.000 99.908 99.584 98.897
75 100.000 100.000 99.908 99.590 98.917
76 100.000 100.000 99.908 99.595 98.936
77 100.000 100.000 99.908 99.599 98.953
78 100.000 100.000 99.908 99.603 98.969
79 100.000 100.000 99.908 99.608 98.988
80 100.000 100.000 99.908 99.614 99.008
81 100.000 100.000 99.908 99.617 99.021
82 100.000 100.000 99.908 99.621 99.037
83 100.000 100.000 99.908 99.625 99.052
84 100.000 100.000 99.908 99.629 99.068
85 100.000 100.000 99.908 99.633 99.084
86 100.000 100.000 99.908 99.637 99.097
87 100.000 100.000 99.908 99.640 99.109
88 100.000 100.000 99.908 99.644 99.124
89 100.000 100.000 99.908 99.647 99.137
90 100.000 100.000 99.908 99.651 99.152
91 100.000 100.000 99.908 99.653 99.161
92 100.000 100.000 99.917 99.680 99.212
93 100.000 100.000 99.917 99.683 99.224
94 100.000 100.000 99.917 99.687 99.236
95 100.000 100.000 99.917 99.689 99.245
96 100.000 100.000 99.917 99.691 99.254
97 100.000 100.000 99.917 99.695 99.269
98 100.000 100.000 99.917 99.697 99.278
99 100.000 100.000 99.917 99.700 99.290
100 100.000 100.000 99.926 99.727 99.341
101 100.000 100.000 99.926 99.729 99.349
102 100.000 100.000 99.935 99.756 99.400
103 100.000 100.000 99.945 99.783 99.450
104 100.000 100.000 99.945 99.785 99.459
105 100.000 100.000 99.954 99.813 99.510
106 100.000 100.000 99.963 99.840 99.560
107 100.000 100.000 99.972 99.867 99.611
108 100.000 100.000 99.982 99.894 99.661
109 100.000 100.000 99.991 99.921 99.712
110 100.000 100.000 100.000 99.948 99.763
111 100.000 100.000 100.000 99.950 99.771
112 100.000 100.000 100.000 99.952 99.779
113 100.000 100.000 100.000 99.954 99.788
114 100.000 100.000 100.000 99.956 99.796
115 100.000 100.000 100.000 99.958 99.804
116 100.000 100.000 100.000 99.959 99.809
117 100.000 100.000 100.000 99.963 99.821
118 100.000 100.000 100.000 99.965 99.829
119 100.000 100.000 100.000 99.967 99.837
120 100.000 100.000 100.000 99.968 99.842
121 100.000 100.000 100.000 99.970 99.850
122 100.000 100.000 100.000 99.972 99.858
123 100.000 100.000 100.000 99.974 99.866
124 100.000 100.000 100.000 99.976 99.874
125 100.000 100.000 100.000 99.978 99.882
126 100.000 100.000 100.000 99.980 99.890
127 100.000 100.000 100.000 99.982 99.898
128 100.000 100.000 100.000 99.984 99.906
129 100.000 100.000 100.000 99.986 99.914
130 100.000 100.000 100.000 99.989 99.922
131 100.000 100.000 100.000 99.990 99.926
132 100.000 100.000 100.000 99.992 99.934
133 100.000 100.000 100.000 99.994 99.942
134 100.000 100.000 100.000 99.996 99.950
135 100.000 100.000 100.000 99.997 99.954
136 100.000 100.000 100.000 99.999 99.962
137 100.000 100.000 100.000 99.999 99.963
138 100.000 100.000 100.000 99.999 99.965
139 100.000 100.000 100.000 99.999 99.966
140 100.000 100.000 100.000 99.999 99.967
141 100.000 100.000 100.000 99.999 99.968
142 100.000 100.000 100.000 99.999 99.969
143 100.000 100.000 100.000 99.999 99.970
144 100.000 100.000 100.000 99.999 99.972
145 100.000 100.000 100.000 99.999 99.973
146 100.000 100.000 100.000 99.999 99.974
147 100.000 100.000 100.000 99.999 99.974
148 100.000 100.000 100.000 99.999 99.975
149 100.000 100.000 100.000 99.999 99.976
150 100.000 100.000 100.000 99.999 99.977
151 100.000 100.000 100.000 99.999 99.978
152 100.000 100.000 100.000 99.999 99.979
153 100.000 100.000 100.000 99.999 99.980
154 100.000 100.000 100.000 99.999 99.981
155 100.000 100.000 100.000 99.999 99.982
156 100.000 100.000 100.000 99.999 99.983
157 100.000 100.000 100.000 100.000 99.987
158 100.000 100.000 100.000 100.000 99.988
159 100.000 100.000 100.000 100.000 99.988
160 100.000 100.000 100.000 100.000 99.989
161 100.000 100.000 100.000 100.000 99.989
162 100.000 100.000 100.000 100.000 99.990
163 100.000 100.000 100.000 100.000 99.991
164 100.000 100.000 100.000 100.000 99.991
165 100.000 100.000 100.000 100.000 99.992
166 100.000 100.000 100.000 100.000 99.993
167 100.000 100.000 100.000 100.000 99.993
168 100.000 100.000 100.000 100.000 99.994
169 100.000 100.000 100.000 100.000 99.994
170 100.000 100.000 100.000 100.000 99.995
171 100.000 100.000 100.000 100.000 99.996
172 100.000 100.000 100.000 100.000 99.996
173 100.000 100.000 100.000 100.000 99.997
174 100.000 100.000 100.000 100.000 99.998
175 100.000 100.000 100.000 100.000 99.998
176 100.000 100.000 100.000 100.000 99.999
177 100.000 100.000 100.000 100.000 99.999
178 100.000 100.000 100.000 100.000 100.000
179 100.000 100.000 100.000 100.000 100.000
180 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 64.286 34.687 19.287 11.372 7.125
4 71.429 41.794 24.402 14.787 9.398
5 80.952 48.900 29.141 17.945 11.545
6 85.714 53.807 32.895 20.632 13.457
7 90.476 58.545 36.696 23.463 15.504
8 95.238 63.283 40.497 26.293 17.551
9 97.619 67.174 44.064 29.071 19.589
10 100.000 70.558 47.208 31.602 21.506
11 100.000 72.927 50.023 34.046 23.407
12 100.000 75.296 52.816 36.481 25.314
13 100.000 77.327 55.256 38.663 27.073
14 100.000 79.188 57.602 40.802 28.803
15 100.000 80.880 59.784 42.840 30.490
16 100.000 82.572 61.872 44.759 32.080
17 100.000 83.926 63.749 46.603 33.661
18 100.000 85.110 65.415 48.287 35.148
19 100.000 86.125 66.753 49.601 36.311
20 100.000 87.140 68.207 51.102 37.667
21 100.000 87.986 69.521 52.517 38.978
22 100.000 88.832 70.695 53.718 40.076
23 100.000 89.679 72.032 55.171 41.426
24 100.000 90.525 73.393 56.662 42.811
25 100.000 91.371 74.801 58.227 44.272
26 100.000 92.047 75.692 59.107 45.058
27 100.000 92.724 76.771 60.308 46.210
28 100.000 93.401 78.038 61.782 47.617
29 100.000 94.078 79.024 62.833 48.612
30 100.000 94.755 80.009 63.884 49.607
31 100.000 95.262 80.784 64.753 50.471
32 100.000 95.770 81.628 65.723 51.429
33 100.000 96.277 82.543 66.822 52.536
34 100.000 96.785 83.388 67.803 53.514
35 100.000 97.293 84.233 68.785 54.496
36 100.000 97.800 85.031 69.680 55.375
37 100.000 98.308 85.734 70.409 56.057
38 100.000 98.646 86.415 71.272 56.963
39 100.000 98.985 87.025 72.034 57.773
40 100.000 99.323 87.541 72.613 58.352
41 100.000 99.662 88.104 73.294 59.073
42 100.000 99.831 88.433 73.717 59.535
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 64.286 34.518 19.193 11.340 7.118
4 71.429 41.455 24.214 14.723 9.383
5 76.190 46.870 28.578 17.881 11.587
6 83.333 52.961 33.060 21.039 13.782
7 85.714 57.191 36.931 24.009 15.922
8 88.095 61.421 40.779 26.958 18.050
9 90.476 65.313 44.392 29.795 20.134
10 92.857 68.697 47.489 32.277 22.022
11 95.238 71.912 50.493 34.738 23.920
12 97.619 74.958 53.332 37.086 25.752
13 97.619 76.819 55.655 39.215 27.492
14 97.619 78.511 57.837 41.263 29.195
15 97.619 80.034 59.808 43.129 30.767
16 97.619 81.049 61.427 44.850 32.313
17 97.619 82.064 62.928 46.432 33.757
18 97.619 83.249 64.524 48.008 35.142
19 97.619 84.264 66.025 49.579 36.565
20 97.619 85.279 67.480 51.091 37.940
21 97.619 86.294 68.888 52.523 39.233
22 97.619 87.310 70.296 53.954 40.526
23 97.619 88.325 71.680 55.338 41.755
24 97.619 88.832 72.712 56.603 43.013
25 97.619 89.509 73.792 57.815 44.185
26 97.619 90.186 74.871 59.016 45.337
27 100.000 92.217 76.420 60.195 46.253
28 100.000 92.893 77.452 61.326 47.338
29 100.000 93.570 78.461 62.410 48.361
30 100.000 94.247 79.446 63.460 49.356
31 100.000 94.924 80.432 64.511 50.350
32 100.000 95.431 81.300 65.530 51.364
33 100.000 95.770 82.074 66.527 52.390
34 100.000 96.108 82.802 67.460 53.363
35 100.000 96.785 83.693 68.339 54.149
36 100.000 97.293 84.491 69.251 55.064
37 100.000 97.800 85.265 70.120 55.928
38 100.000 98.308 86.016 70.945 56.739
39 100.000 98.646 86.649 71.733 57.566
40 100.000 99.154 87.353 72.463 58.248
41 100.000 99.492 87.916 73.138 58.957
42 100.000 99.831 88.433 73.717 59.535
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 61.905 33.503 18.841 11.233 7.090
4 71.429 41.455 24.261 14.771 9.413
5 80.952 49.239 29.517 18.208 11.687
6 85.714 54.653 33.881 21.366 13.891
7 88.095 59.391 38.104 24.503 16.094
8 90.476 63.621 41.952 27.452 18.222
9 90.476 65.990 44.721 29.864 20.118
10 92.857 68.866 47.583 32.309 22.030
11 95.238 71.912 50.446 34.684 23.881
12 100.000 76.142 53.778 37.221 25.778
13 100.000 77.834 55.913 39.210 27.435
14 100.000 79.357 57.907 41.113 29.044
15 100.000 80.372 59.573 42.866 30.592
16 100.000 81.387 61.098 44.480 32.062
17 100.000 82.741 62.905 46.239 33.578
18 100.000 83.926 64.547 47.896 35.046
19 100.000 84.941 66.002 49.408 36.422
20 100.000 85.787 67.316 50.839 37.761
21 100.000 86.802 68.771 52.340 39.116
22 100.000 87.986 70.366 53.917 40.502
23 100.000 88.832 71.633 55.289 41.794
24 100.000 89.848 73.041 56.715 43.079
25 100.000 90.863 74.449 58.142 44.365
26 100.000 91.709 75.669 59.423 45.554
27 100.000 92.724 77.030 60.774 46.756
28 100.000 93.401 78.109 61.975 47.908
29 100.000 94.078 79.141 63.107 48.993
30 100.000 94.416 79.939 64.131 50.038
31 100.000 95.093 80.924 65.181 51.033
32 100.000 95.601 81.746 66.120 51.963
33 100.000 95.939 82.426 66.983 52.869
34 100.000 96.447 83.247 67.921 53.799
35 100.000 96.954 84.045 68.822 54.691
36 100.000 97.462 84.819 69.691 55.554
37 100.000 98.139 85.711 70.570 56.340
38 100.000 98.477 86.274 71.251 57.061
39 100.000 98.816 86.861 71.948 57.773
40 100.000 99.323 87.565 72.677 58.456
41 100.000 99.662 88.081 73.256 59.034
42 100.000 99.831 88.409 73.680 59.497
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 59.524 32.318 18.348 11.045 7.025
4 66.667 39.594 23.627 14.562 9.346
5 73.810 46.701 28.789 18.026 11.648
6 78.571 51.777 32.895 21.050 13.797
7 80.952 54.992 36.039 23.666 15.800
8 83.333 58.376 39.207 26.235 17.752
9 88.095 62.944 42.891 29.012 19.781
10 88.095 65.144 45.519 31.333 21.623
11 88.095 67.343 48.170 33.676 23.476
12 90.476 70.051 50.634 35.757 25.159
13 92.857 73.096 53.426 38.046 26.945
14 92.857 74.450 55.256 39.842 28.500
15 97.619 78.680 58.517 42.282 30.312
16 97.619 80.034 60.347 44.078 31.864
17 100.000 82.910 62.905 46.164 33.510
18 100.000 84.095 64.547 47.810 34.961
19 100.000 84.941 65.931 49.322 36.357
20 100.000 86.125 67.597 50.995 37.828
21 100.000 86.971 68.935 52.448 39.177
22 100.000 87.817 70.202 53.809 40.453
23 100.000 88.832 71.656 55.311 41.809
24 100.000 90.017 73.229 56.855 43.166
25 100.000 90.863 74.519 58.238 44.450
26 100.000 91.878 75.880 59.611 45.696
27 100.000 92.555 76.983 60.844 46.876
28 100.000 93.401 78.179 62.088 48.026
29 100.000 94.078 79.165 63.155 49.056
30 100.000 94.416 79.962 64.179 50.101
31 100.000 94.755 80.666 65.069 51.022
32 100.000 95.262 81.488 66.007 51.952
33 100.000 95.770 82.309 66.945 52.882
34 100.000 96.447 83.271 67.959 53.838
35 100.000 97.124 84.233 68.972 54.794
36 100.000 97.631 85.007 69.841 55.658
37 100.000 97.970 85.570 70.522 56.379
38 100.000 98.308 86.157 71.219 57.091
39 100.000 98.816 86.861 71.948 57.773
40 100.000 99.323 87.565 72.677 58.456
41 100.000 99.662 88.081 73.256 59.034
42 100.000 99.831 88.409 73.680 59.497
43 100.000 100.000 88.714 74.066 59.920

grep

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 66.667 35.533 19.521 11.426 7.134
4 76.190 43.486 24.941 14.964 9.457
5 85.714 51.100 30.103 18.369 11.724
6 90.476 57.191 34.937 21.747 14.010
7 92.857 61.929 39.160 24.883 16.212
8 95.238 66.328 43.149 27.913 18.377
9 97.619 70.051 46.574 30.615 20.386
10 100.000 73.435 49.789 33.242 22.385
11 100.000 75.296 52.112 35.376 24.134
12 100.000 77.157 54.575 37.671 25.996
13 100.000 79.019 56.875 39.778 27.727
14 100.000 80.541 59.033 41.880 29.488
15 100.000 81.895 61.004 43.842 31.165
16 100.000 83.249 62.764 45.536 32.626
17 100.000 84.433 64.406 47.182 34.077
18 100.000 85.448 65.814 48.619 35.379
19 100.000 86.464 67.222 50.051 36.672
20 100.000 87.479 68.583 51.402 37.874
21 100.000 88.494 70.014 52.866 39.194
22 100.000 89.340 71.258 54.179 40.411
23 100.000 90.186 72.642 55.713 41.844
24 100.000 91.032 74.050 57.257 43.268
25 100.000 91.709 75.364 58.774 44.692
26 100.000 92.386 76.396 59.895 45.753
27 100.000 93.063 77.288 60.774 46.539
28 100.000 93.570 78.179 61.847 47.628
29 100.000 94.078 78.954 62.715 48.492
30 100.000 94.585 79.798 63.707 49.494
31 100.000 95.093 80.526 64.474 50.215
32 100.000 95.601 81.253 65.240 50.937
33 100.000 96.108 82.074 66.179 51.867
34 100.000 96.447 82.708 66.967 52.693
35 100.000 96.785 83.388 67.825 53.587
36 100.000 97.124 83.881 68.361 54.114
37 100.000 97.462 84.420 69.004 54.796
38 100.000 97.800 85.031 69.771 55.615
39 100.000 97.970 85.617 70.629 56.555
40 100.000 98.139 86.133 71.374 57.383
41 100.000 98.308 86.509 71.894 57.977
42 100.000 98.477 86.908 72.452 58.609
43 100.000 98.646 87.353 73.101 59.352
44 100.000 98.816 87.682 73.540 59.848
45 100.000 98.985 88.034 74.007 60.362
46 100.000 99.154 88.409 74.532 60.967
47 100.000 99.323 88.761 75.031 61.553
48 100.000 99.492 89.254 75.739 62.343
49 100.000 99.662 89.653 76.296 62.975
50 100.000 99.831 89.911 76.591 63.280
51 100.000 100.000 90.216 76.977 63.704
52 100.000 100.000 90.357 77.261 64.090
53 100.000 100.000 90.802 78.055 65.031
54 100.000 100.000 90.990 78.441 65.553
55 100.000 100.000 91.061 78.586 65.754
56 100.000 100.000 91.107 78.687 65.904
57 100.000 100.000 91.248 78.961 66.263
58 100.000 100.000 91.436 79.336 66.764
59 100.000 100.000 91.741 79.888 67.437
60 100.000 100.000 91.929 80.264 67.938
61 100.000 100.000 92.046 80.505 68.269
62 100.000 100.000 92.281 80.961 68.861
63 100.000 100.000 92.374 81.165 69.153
64 100.000 100.000 92.562 81.545 69.667
65 100.000 100.000 92.679 81.781 69.987
66 100.000 100.000 92.726 81.883 70.137
67 100.000 100.000 92.844 82.130 70.485
68 100.000 100.000 92.867 82.189 70.583
69 100.000 100.000 92.914 82.290 70.732
70 100.000 100.000 92.985 82.441 70.949
71 100.000 100.000 93.102 82.682 71.284
72 100.000 100.000 93.172 82.843 71.524
73 100.000 100.000 93.290 83.084 71.855
74 100.000 100.000 93.407 83.320 72.175
75 100.000 100.000 93.454 83.427 72.339
76 100.000 100.000 93.595 83.722 72.750
77 100.000 100.000 93.688 83.915 73.018
78 100.000 100.000 93.759 84.065 73.234
79 100.000 100.000 93.782 84.119 73.318
80 100.000 100.000 94.017 84.521 73.789
81 100.000 100.000 94.111 84.714 74.057
82 100.000 100.000 94.392 85.245 74.727
83 100.000 100.000 94.674 85.781 75.398
84 100.000 100.000 94.721 85.904 75.601
85 100.000 100.000 94.791 86.076 75.873
86 100.000 100.000 94.979 86.462 76.395
87 100.000 100.000 95.190 86.880 76.940
88 100.000 100.000 95.237 87.014 77.171
89 100.000 100.000 95.331 87.229 77.491
90 100.000 100.000 95.448 87.464 77.813
91 100.000 100.000 95.565 87.711 78.161
92 100.000 100.000 95.636 87.856 78.362
93 100.000 100.000 95.753 88.113 78.734
94 100.000 100.000 95.847 88.306 79.002
95 100.000 100.000 95.870 88.381 79.141
96 100.000 100.000 95.964 88.590 79.448
97 100.000 100.000 95.988 88.671 79.599
98 100.000 100.000 96.058 88.832 79.844
99 100.000 100.000 96.293 89.277 80.411
100 100.000 100.000 96.387 89.491 80.724
101 100.000 100.000 96.457 89.647 80.953
102 100.000 100.000 96.621 89.990 81.423
103 100.000 100.000 96.809 90.370 81.937
104 100.000 100.000 96.832 90.424 82.020
105 100.000 100.000 96.973 90.698 82.379
106 100.000 100.000 97.255 91.212 83.013
107 100.000 100.000 97.349 91.405 83.281
108 100.000 100.000 97.396 91.507 83.431
109 100.000 100.000 97.489 91.722 83.751
110 100.000 100.000 97.630 92.022 84.174
111 100.000 100.000 97.724 92.220 84.455
112 100.000 100.000 97.748 92.285 84.566
113 100.000 100.000 97.794 92.386 84.716
114 100.000 100.000 97.888 92.590 85.012
115 100.000 100.000 97.912 92.649 85.112
116 100.000 100.000 97.959 92.762 85.289
117 100.000 100.000 98.053 92.955 85.558
118 100.000 100.000 98.076 93.025 85.682
119 100.000 100.000 98.170 93.228 85.977
120 100.000 100.000 98.264 93.421 86.246
121 100.000 100.000 98.358 93.625 86.534
122 100.000 100.000 98.381 93.738 86.761
123 100.000 100.000 98.405 93.797 86.859
124 100.000 100.000 98.451 93.898 87.009
125 100.000 100.000 98.522 94.054 87.239
126 100.000 100.000 98.710 94.429 87.740
127 100.000 100.000 98.756 94.531 87.889
128 100.000 100.000 98.850 94.724 88.158
129 100.000 100.000 98.921 94.874 88.374
130 100.000 100.000 98.968 94.976 88.524
131 100.000 100.000 99.085 95.233 88.895
132 100.000 100.000 99.108 95.303 89.019
133 100.000 100.000 99.179 95.459 89.249
134 100.000 100.000 99.202 95.512 89.332
135 100.000 100.000 99.249 95.646 89.563
136 100.000 100.000 99.273 95.711 89.674
137 100.000 100.000 99.320 95.823 89.852
138 100.000 100.000 99.413 96.016 90.120
139 100.000 100.000 99.460 96.129 90.297
140 100.000 100.000 99.484 96.193 90.410
141 100.000 100.000 99.578 96.397 90.706
142 100.000 100.000 99.648 96.552 90.935
143 100.000 100.000 99.695 96.708 91.213
144 100.000 100.000 99.742 96.810 91.362
145 100.000 100.000 99.812 96.981 91.632
146 100.000 100.000 99.883 97.132 91.849
147 100.000 100.000 99.906 97.201 91.974
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 61.905 33.841 19.052 11.318 7.116
4 73.810 42.640 24.707 14.911 9.448
5 80.952 49.915 29.986 18.428 11.768
6 83.333 54.146 33.904 21.447 13.938
7 85.714 57.868 37.424 24.251 16.014
8 92.857 63.621 41.600 27.216 18.108
9 95.238 67.343 45.002 29.891 20.097
10 95.238 69.543 47.701 32.293 21.996
11 100.000 73.942 51.103 34.813 23.846
12 100.000 76.142 53.754 37.156 25.699
13 100.000 78.173 56.288 39.456 27.550
14 100.000 80.203 58.799 41.735 29.390
15 100.000 81.557 60.770 43.697 31.067
16 100.000 82.910 62.694 45.611 32.714
17 100.000 84.095 64.313 47.236 34.155
18 100.000 85.448 66.002 48.812 35.486
19 100.000 86.464 67.480 50.362 36.899
20 100.000 87.479 68.935 51.863 38.255
21 100.000 88.325 70.272 53.316 39.605
22 100.000 89.171 71.563 54.699 40.888
23 100.000 90.017 72.853 56.104 42.208
24 100.000 90.863 74.097 57.439 43.462
25 100.000 91.709 75.340 58.758 44.688
26 100.000 92.386 76.466 60.008 45.862
27 100.000 93.063 77.546 61.209 47.014
28 100.000 93.739 78.531 62.286 48.065
29 100.000 94.247 79.423 63.326 49.089
30 100.000 94.247 80.033 64.350 50.234
31 100.000 94.755 80.924 65.412 51.303
32 100.000 95.262 81.793 66.420 52.299
33 100.000 95.431 82.450 67.374 53.331
34 100.000 95.939 83.294 68.361 54.324
35 100.000 96.447 84.092 69.283 55.260
36 100.000 96.954 84.866 70.146 56.113
37 100.000 97.293 85.547 71.004 57.006
38 100.000 97.462 86.110 71.830 57.921
39 100.000 97.800 86.720 72.581 58.709
40 100.000 97.800 87.142 73.315 59.567
41 100.000 98.139 87.752 74.066 60.355
42 100.000 98.139 88.104 74.704 61.115
43 100.000 98.139 88.456 75.331 61.867
44 100.000 98.308 88.878 75.932 62.550
45 100.000 98.646 89.371 76.500 63.153
46 100.000 98.816 89.770 77.063 63.796
47 100.000 98.985 90.145 77.583 64.390
48 100.000 99.154 90.521 78.103 64.983
49 100.000 99.323 90.896 78.623 65.577
50 100.000 99.323 91.154 79.116 66.207
51 100.000 99.492 91.506 79.615 66.794
52 100.000 99.492 91.741 80.065 67.372
53 100.000 99.492 91.976 80.500 67.916
54 100.000 99.662 92.304 80.950 68.439
55 100.000 99.831 92.633 81.374 68.902
56 100.000 99.831 92.867 81.835 69.505
57 100.000 99.831 93.078 82.253 70.050
58 100.000 99.831 93.266 82.628 70.551
59 100.000 99.831 93.454 82.998 71.041
60 100.000 99.831 93.641 83.395 71.591
61 100.000 99.831 93.829 83.770 72.092
62 100.000 99.831 94.017 84.151 72.606
63 100.000 100.000 94.275 84.446 72.911
64 100.000 100.000 94.439 84.784 73.373
65 100.000 100.000 94.603 85.100 73.784
66 100.000 100.000 94.744 85.384 74.170
67 100.000 100.000 94.885 85.700 74.625
68 100.000 100.000 95.026 85.974 74.984
69 100.000 100.000 95.167 86.253 75.356
70 100.000 100.000 95.307 86.553 75.777
71 100.000 100.000 95.448 86.837 76.156
72 100.000 100.000 95.589 87.121 76.535
73 100.000 100.000 95.730 87.400 76.907
74 100.000 100.000 95.870 87.695 77.318
75 100.000 100.000 96.011 88.001 77.752
76 100.000 100.000 96.129 88.253 78.111
77 100.000 100.000 96.246 88.483 78.418
78 100.000 100.000 96.363 88.719 78.738
79 100.000 100.000 96.481 88.976 79.111
80 100.000 100.000 96.574 89.180 79.405
81 100.000 100.000 96.668 89.373 79.673
82 100.000 100.000 96.762 89.641 80.110
83 100.000 100.000 96.856 89.872 80.467
84 100.000 100.000 96.950 90.070 80.750
85 100.000 100.000 97.044 90.269 81.031
86 100.000 100.000 97.137 90.499 81.388
87 100.000 100.000 97.231 90.714 81.708
88 100.000 100.000 97.302 90.864 81.924
89 100.000 100.000 97.372 91.009 82.125
90 100.000 100.000 97.443 91.175 82.379
91 100.000 100.000 97.513 91.357 82.671
92 100.000 100.000 97.583 91.534 82.952
93 100.000 100.000 97.654 91.700 83.205
94 100.000 100.000 97.724 91.856 83.435
95 100.000 100.000 97.794 92.017 83.679
96 100.000 100.000 97.865 92.188 83.949
97 100.000 100.000 97.935 92.360 84.219
98 100.000 100.000 98.006 92.510 84.435
99 100.000 100.000 98.076 92.660 84.652
100 100.000 100.000 98.146 92.815 84.881
101 100.000 100.000 98.217 92.966 85.097
102 100.000 100.000 98.287 93.148 85.391
103 100.000 100.000 98.358 93.346 85.722
104 100.000 100.000 98.428 93.502 85.952
105 100.000 100.000 98.475 93.604 86.101
106 100.000 100.000 98.522 93.722 86.294
107 100.000 100.000 98.569 93.829 86.458
108 100.000 100.000 98.616 93.931 86.608
109 100.000 100.000 98.663 94.049 86.798
110 100.000 100.000 98.710 94.156 86.963
111 100.000 100.000 98.756 94.268 87.141
112 100.000 100.000 98.803 94.381 87.322
113 100.000 100.000 98.850 94.504 87.527
114 100.000 100.000 98.897 94.606 87.677
115 100.000 100.000 98.944 94.708 87.827
116 100.000 100.000 98.991 94.810 87.976
117 100.000 100.000 99.038 94.923 88.154
118 100.000 100.000 99.085 95.030 88.317
119 100.000 100.000 99.132 95.137 88.481
120 100.000 100.000 99.179 95.239 88.631
121 100.000 100.000 99.226 95.394 88.908
122 100.000 100.000 99.273 95.518 89.110
123 100.000 100.000 99.320 95.636 89.302
124 100.000 100.000 99.366 95.737 89.452
125 100.000 100.000 99.413 95.861 89.654
126 100.000 100.000 99.460 95.968 89.818
127 100.000 100.000 99.507 96.086 90.007
128 100.000 100.000 99.531 96.156 90.131
129 100.000 100.000 99.554 96.225 90.256
130 100.000 100.000 99.578 96.279 90.340
131 100.000 100.000 99.601 96.343 90.454
132 100.000 100.000 99.625 96.429 90.621
133 100.000 100.000 99.648 96.488 90.719
134 100.000 100.000 99.672 96.542 90.802
135 100.000 100.000 99.695 96.601 90.900
136 100.000 100.000 99.718 96.660 90.998
137 100.000 100.000 99.742 96.719 91.096
138 100.000 100.000 99.765 96.772 91.179
139 100.000 100.000 99.789 96.837 91.290
140 100.000 100.000 99.812 96.890 91.373
141 100.000 100.000 99.836 96.955 91.486
142 100.000 100.000 99.859 97.014 91.586
143 100.000 100.000 99.883 97.110 91.778
144 100.000 100.000 99.906 97.164 91.862
145 100.000 100.000 99.930 97.223 91.960
146 100.000 100.000 99.953 97.282 92.058
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 57.143 31.641 18.254 11.072 7.053
4 66.667 39.425 23.557 14.557 9.357
5 76.190 47.208 28.860 18.042 11.661
6 80.952 52.623 33.224 21.200 13.865
7 85.714 57.530 37.166 24.122 15.966
8 88.095 61.083 40.544 26.851 18.013
9 90.476 64.129 43.548 29.382 19.970
10 90.476 66.328 46.293 31.832 21.898
11 95.238 71.066 50.047 34.615 23.916
12 97.619 74.281 53.050 37.076 25.813
13 97.619 76.142 55.397 39.237 27.581
14 97.619 77.834 57.626 41.344 29.330
15 100.000 80.711 60.300 43.590 31.115
16 100.000 82.403 62.412 45.542 32.733
17 100.000 83.249 63.820 47.113 34.201
18 100.000 84.772 65.767 48.952 35.755
19 100.000 85.956 67.433 50.630 37.234
20 100.000 87.310 69.193 52.319 38.684
21 100.000 88.494 70.765 53.868 40.050
22 100.000 89.171 71.938 55.209 41.336
23 100.000 90.017 73.205 56.565 42.600
24 100.000 90.863 74.449 57.884 43.828
25 100.000 91.540 75.575 59.134 45.002
26 100.000 92.047 76.513 60.265 46.138
27 100.000 92.893 77.710 61.498 47.264
28 100.000 93.232 78.461 62.474 48.287
29 100.000 93.570 79.235 63.471 49.309
30 100.000 94.078 80.103 64.490 50.323
31 100.000 94.416 80.807 65.412 51.310
32 100.000 94.755 81.511 66.334 52.283
33 100.000 95.093 82.168 67.192 53.213
34 100.000 95.262 82.708 67.986 54.102
35 100.000 95.431 83.200 68.709 54.927
36 100.000 95.939 83.951 69.551 55.772
37 100.000 96.447 84.702 70.388 56.608
38 100.000 96.447 85.031 70.994 57.353
39 100.000 96.616 85.500 71.664 58.104
40 100.000 96.785 85.969 72.334 58.853
41 100.000 97.124 86.579 73.079 59.628
42 100.000 97.293 87.025 73.728 60.371
43 100.000 97.462 87.471 74.366 61.094
44 100.000 97.800 88.034 75.058 61.840
45 100.000 97.800 88.315 75.594 62.511
46 100.000 97.970 88.738 76.200 63.206
47 100.000 98.139 89.137 76.779 63.879
48 100.000 98.646 89.864 77.545 64.600
49 100.000 98.816 90.263 78.114 65.257
50 100.000 98.985 90.662 78.671 65.889
51 100.000 99.154 91.061 79.229 66.522
52 100.000 99.154 91.295 79.685 67.114
53 100.000 99.154 91.530 80.140 67.705
54 100.000 99.154 91.741 80.559 68.251
55 100.000 99.154 91.952 80.972 68.791
56 100.000 99.154 92.140 81.363 69.327
57 100.000 99.154 92.281 81.701 69.830
58 100.000 99.154 92.468 82.076 70.331
59 100.000 99.154 92.656 82.451 70.832
60 100.000 99.154 92.820 82.800 71.319
61 100.000 99.154 93.031 83.191 71.807
62 100.000 99.323 93.360 83.631 72.302
63 100.000 99.492 93.688 84.071 72.805
64 100.000 99.492 93.853 84.414 73.279
65 100.000 99.492 94.017 84.751 73.741
66 100.000 99.492 94.181 85.089 74.203
67 100.000 99.662 94.486 85.497 74.671
68 100.000 99.662 94.627 85.808 75.119
69 100.000 99.662 94.791 86.135 75.557
70 100.000 99.662 94.908 86.408 75.964
71 100.000 99.662 95.073 86.725 76.374
72 100.000 99.662 95.214 87.019 76.785
73 100.000 99.662 95.354 87.314 77.197
74 100.000 99.662 95.495 87.604 77.595
75 100.000 99.831 95.800 87.990 78.018
76 100.000 100.000 96.105 88.376 78.442
77 100.000 100.000 96.246 88.660 78.828
78 100.000 100.000 96.340 88.896 79.193
79 100.000 100.000 96.457 89.148 79.553
80 100.000 100.000 96.551 89.368 79.888
81 100.000 100.000 96.668 89.609 80.221
82 100.000 100.000 96.786 89.850 80.556
83 100.000 100.000 96.856 90.038 80.861
84 100.000 100.000 96.950 90.247 81.169
85 100.000 100.000 97.044 90.451 81.458
86 100.000 100.000 97.137 90.655 81.750
87 100.000 100.000 97.231 90.858 82.048
88 100.000 100.000 97.325 91.057 82.329
89 100.000 100.000 97.419 91.250 82.597
90 100.000 100.000 97.489 91.421 82.865
91 100.000 100.000 97.560 91.593 83.133
92 100.000 100.000 97.630 91.759 83.390
93 100.000 100.000 97.701 91.925 83.647
94 100.000 100.000 97.771 92.086 83.890
95 100.000 100.000 97.841 92.247 84.130
96 100.000 100.000 97.912 92.403 84.359
97 100.000 100.000 97.982 92.558 84.592
98 100.000 100.000 98.053 92.714 84.822
99 100.000 100.000 98.099 92.842 85.040
100 100.000 100.000 98.170 92.992 85.258
101 100.000 100.000 98.240 93.142 85.474
102 100.000 100.000 98.311 93.293 85.691
103 100.000 100.000 98.381 93.443 85.907
104 100.000 100.000 98.451 93.587 86.109
105 100.000 100.000 98.498 93.711 86.310
106 100.000 100.000 98.545 93.834 86.516
107 100.000 100.000 98.592 93.957 86.717
108 100.000 100.000 98.639 94.075 86.906
109 100.000 100.000 98.663 94.172 87.098
110 100.000 100.000 98.710 94.284 87.276
111 100.000 100.000 98.756 94.397 87.455
112 100.000 100.000 98.803 94.510 87.632
113 100.000 100.000 98.850 94.622 87.810
114 100.000 100.000 98.897 94.735 87.989
115 100.000 100.000 98.944 94.847 88.167
116 100.000 100.000 98.991 94.960 88.344
117 100.000 100.000 99.015 95.046 88.513
118 100.000 100.000 99.061 95.153 88.679
119 100.000 100.000 99.108 95.260 88.844
120 100.000 100.000 99.155 95.362 88.993
121 100.000 100.000 99.202 95.464 89.143
122 100.000 100.000 99.249 95.566 89.293
123 100.000 100.000 99.296 95.668 89.443
124 100.000 100.000 99.343 95.770 89.593
125 100.000 100.000 99.390 95.872 89.742
126 100.000 100.000 99.437 95.973 89.892
127 100.000 100.000 99.484 96.075 90.042
128 100.000 100.000 99.531 96.177 90.194
129 100.000 100.000 99.554 96.252 90.332
130 100.000 100.000 99.578 96.327 90.471
131 100.000 100.000 99.601 96.397 90.597
132 100.000 100.000 99.625 96.467 90.722
133 100.000 100.000 99.648 96.536 90.846
134 100.000 100.000 99.672 96.601 90.957
135 100.000 100.000 99.695 96.665 91.068
136 100.000 100.000 99.718 96.729 91.179
137 100.000 100.000 99.742 96.794 91.292
138 100.000 100.000 99.765 96.858 91.405
139 100.000 100.000 99.789 96.922 91.516
140 100.000 100.000 99.812 96.981 91.614
141 100.000 100.000 99.836 97.035 91.697
142 100.000 100.000 99.859 97.089 91.780
143 100.000 100.000 99.883 97.142 91.863
144 100.000 100.000 99.906 97.196 91.947
145 100.000 100.000 99.930 97.249 92.030
146 100.000 100.000 99.953 97.303 92.113
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 42.857 21.658 12.107 7.292 4.656
3 52.381 29.272 17.269 10.696 6.923
4 59.524 36.041 22.220 14.074 9.200
5 71.429 44.501 27.616 17.533 11.476
6 78.571 51.100 32.426 20.830 13.716
7 80.952 56.176 36.861 24.052 15.944
8 85.714 61.421 41.107 27.157 18.130
9 90.476 66.159 44.932 30.025 20.212
10 95.238 70.728 48.545 32.727 22.192
11 95.238 73.096 51.337 35.162 24.099
12 97.619 76.311 54.411 37.698 26.044
13 100.000 79.188 57.109 39.966 27.840
14 100.000 80.372 58.893 41.810 29.467
15 100.000 81.726 60.863 43.772 31.144
16 100.000 83.080 62.740 45.633 32.755
17 100.000 84.095 64.336 47.327 34.283
18 100.000 85.618 66.283 49.166 35.836
19 100.000 86.633 67.785 50.732 37.251
20 100.000 87.648 69.310 52.324 38.684
21 100.000 88.325 70.507 53.697 39.997
22 100.000 89.509 72.055 55.214 41.336
23 100.000 90.186 73.158 56.453 42.526
24 100.000 90.694 74.097 57.579 43.653
25 100.000 91.371 75.153 58.748 44.777
26 100.000 92.047 76.208 59.916 45.901
27 100.000 92.555 77.100 60.978 46.970
28 100.000 92.724 77.687 61.852 47.946
29 100.000 93.232 78.578 62.903 48.995
30 100.000 93.739 79.446 63.922 50.016
31 100.000 94.078 80.150 64.828 50.972
32 100.000 94.585 80.948 65.750 51.907
33 100.000 94.755 81.534 66.608 52.847
34 100.000 94.755 81.863 67.230 53.627
35 100.000 95.431 82.872 68.313 54.650
36 100.000 95.601 83.365 69.037 55.471
37 100.000 95.939 84.045 69.894 56.364
38 100.000 95.939 84.374 70.500 57.109
39 100.000 96.277 85.007 71.294 57.947
40 100.000 96.785 85.805 72.195 58.838
41 100.000 97.124 86.415 72.951 59.637
42 100.000 97.631 87.213 73.851 60.525
43 100.000 97.800 87.635 74.468 61.240
44 100.000 97.970 88.081 75.111 61.975
45 100.000 97.970 88.339 75.610 62.616
46 100.000 97.970 88.620 76.135 63.275
47 100.000 97.970 88.902 76.671 63.946
48 100.000 98.139 89.301 77.245 64.613
49 100.000 98.308 89.700 77.814 65.270
50 100.000 98.477 90.122 78.409 65.941
51 100.000 98.646 90.521 78.972 66.585
52 100.000 98.646 90.756 79.422 67.163
53 100.000 98.646 90.967 79.840 67.716
54 100.000 98.816 91.366 80.398 68.349
55 100.000 98.816 91.577 80.811 68.891
56 100.000 98.816 91.788 81.224 69.432
57 100.000 98.816 91.976 81.610 69.954
58 100.000 98.816 92.163 81.985 70.455
59 100.000 98.816 92.328 82.328 70.928
60 100.000 98.985 92.679 82.811 71.483
61 100.000 98.985 92.867 83.181 71.973
62 100.000 98.985 93.055 83.545 72.454
63 100.000 99.154 93.360 83.958 72.942
64 100.000 99.154 93.501 84.258 73.366
65 100.000 99.323 93.829 84.698 73.869
66 100.000 99.323 93.993 85.025 74.307
67 100.000 99.492 94.298 85.432 74.782
68 100.000 99.492 94.439 85.727 75.193
69 100.000 99.492 94.580 86.022 75.603
70 100.000 99.492 94.721 86.317 76.014
71 100.000 99.492 94.862 86.607 76.413
72 100.000 99.662 95.190 87.046 76.909
73 100.000 99.662 95.331 87.330 77.297
74 100.000 99.662 95.495 87.652 77.719
75 100.000 99.662 95.612 87.904 78.079
76 100.000 99.662 95.730 88.151 78.427
77 100.000 99.662 95.824 88.365 78.747
78 100.000 99.662 95.941 88.612 79.093
79 100.000 99.662 96.035 88.826 79.413
80 100.000 99.662 96.176 89.111 79.792
81 100.000 99.662 96.269 89.320 80.102
82 100.000 99.662 96.387 89.561 80.437
83 100.000 99.662 96.481 89.770 80.746
84 100.000 99.662 96.574 89.979 81.053
85 100.000 99.662 96.668 90.183 81.349
86 100.000 99.662 96.762 90.387 81.645
87 100.000 99.662 96.856 90.596 81.948
88 100.000 99.662 96.950 90.799 82.244
89 100.000 99.662 97.044 91.003 82.540
90 100.000 99.662 97.091 91.143 82.780
91 100.000 99.831 97.372 91.486 83.152
92 100.000 99.831 97.466 91.689 83.440
93 100.000 99.831 97.489 91.786 83.634
94 100.000 99.831 97.536 91.915 83.853
95 100.000 99.831 97.560 92.011 84.045
96 100.000 99.831 97.630 92.172 84.289
97 100.000 99.831 97.724 92.365 84.557
98 100.000 99.831 97.794 92.521 84.790
99 100.000 99.831 97.841 92.644 84.995
100 100.000 99.831 97.912 92.799 85.225
101 100.000 99.831 97.982 92.955 85.454
102 100.000 99.831 98.053 93.110 85.683
103 100.000 99.831 98.099 93.234 85.885
104 100.000 99.831 98.146 93.357 86.087
105 100.000 99.831 98.193 93.475 86.279
106 100.000 99.831 98.217 93.561 86.445
107 100.000 99.831 98.287 93.711 86.662
108 100.000 99.831 98.358 93.861 86.878
109 100.000 99.831 98.428 94.011 87.094
110 100.000 100.000 98.686 94.306 87.399
111 100.000 100.000 98.756 94.456 87.616
112 100.000 100.000 98.803 94.574 87.804
113 100.000 100.000 98.827 94.654 87.958
114 100.000 100.000 98.874 94.767 88.135
115 100.000 100.000 98.921 94.880 88.313
116 100.000 100.000 98.968 94.992 88.490
117 100.000 100.000 99.038 95.137 88.692
118 100.000 100.000 99.085 95.244 88.858
119 100.000 100.000 99.132 95.351 89.023
120 100.000 100.000 99.179 95.453 89.175
121 100.000 100.000 99.202 95.523 89.300
122 100.000 100.000 99.249 95.625 89.450
123 100.000 100.000 99.273 95.695 89.574
124 100.000 100.000 99.320 95.796 89.724
125 100.000 100.000 99.366 95.898 89.874
126 100.000 100.000 99.413 96.000 90.023
127 100.000 100.000 99.460 96.102 90.173
128 100.000 100.000 99.507 96.204 90.323
129 100.000 100.000 99.531 96.274 90.447
130 100.000 100.000 99.578 96.376 90.597
131 100.000 100.000 99.601 96.440 90.710
132 100.000 100.000 99.625 96.504 90.820
133 100.000 100.000 99.648 96.569 90.931
134 100.000 100.000 99.672 96.633 91.042
135 100.000 100.000 99.695 96.697 91.153
136 100.000 100.000 99.718 96.756 91.251
137 100.000 100.000 99.742 96.815 91.349
138 100.000 100.000 99.765 96.874 91.447
139 100.000 100.000 99.789 96.928 91.531
140 100.000 100.000 99.812 96.981 91.614
141 100.000 100.000 99.836 97.035 91.697
142 100.000 100.000 99.859 97.089 91.780
143 100.000 100.000 99.883 97.142 91.863
144 100.000 100.000 99.906 97.196 91.947
145 100.000 100.000 99.930 97.249 92.030
146 100.000 100.000 99.953 97.303 92.113
147 100.000 100.000 99.977 97.357 92.196
148 100.000 100.000 100.000 97.410 92.280

grep

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 66.667 35.533 19.521 11.426 7.134
4 76.190 43.486 24.941 14.964 9.457
5 80.952 49.577 29.775 18.342 11.743
6 85.714 55.668 34.608 21.720 14.028
7 92.857 62.098 39.301 24.964 16.249
8 95.238 66.836 43.524 28.100 18.451
9 97.619 70.389 46.715 30.594 20.310
10 100.000 73.435 49.648 33.049 22.218
11 100.000 75.635 52.229 35.301 23.994
12 100.000 77.496 54.364 37.151 25.464
13 100.000 79.357 56.617 39.183 27.119
14 100.000 81.049 58.611 40.952 28.552
15 100.000 82.572 60.676 42.947 30.237
16 100.000 83.926 62.694 44.957 31.943
17 100.000 85.279 64.735 46.989 33.661
18 100.000 86.464 66.448 48.737 35.207
19 100.000 87.648 68.067 50.319 36.566
20 100.000 88.663 69.451 51.702 37.796
21 100.000 89.509 70.695 53.016 39.013
22 100.000 90.355 72.102 54.560 40.437
23 100.000 91.201 73.322 55.836 41.615
24 100.000 92.047 74.542 57.107 42.780
25 100.000 92.893 75.809 58.426 43.974
26 100.000 93.570 76.795 59.466 44.945
27 100.000 94.247 77.945 60.742 46.138
28 100.000 94.755 78.672 61.509 46.859
29 100.000 95.262 79.587 62.619 47.983
30 100.000 95.770 80.432 63.616 48.986
31 100.000 96.108 80.995 64.286 49.683
32 100.000 96.447 81.699 65.181 50.613
33 100.000 96.785 82.426 66.131 51.617
34 100.000 97.124 82.942 66.715 52.207
35 100.000 97.462 83.458 67.294 52.786
36 100.000 97.800 84.186 68.222 53.746
37 100.000 98.139 85.007 69.267 54.798
38 100.000 98.477 85.687 70.141 55.726
39 100.000 98.816 86.204 70.720 56.305
40 100.000 99.154 86.861 71.546 57.163
41 100.000 99.323 87.166 71.932 57.586
42 100.000 99.492 87.494 72.361 58.062
43 100.000 99.662 88.175 73.379 59.179
44 100.000 99.831 88.503 73.803 59.641
45 100.000 100.000 88.832 74.243 60.136
46 100.000 100.000 89.043 74.650 60.665
47 100.000 100.000 89.230 75.015 61.142
48 100.000 100.000 89.559 75.631 61.904
49 100.000 100.000 89.747 76.007 62.405
50 100.000 100.000 89.840 76.205 62.687
51 100.000 100.000 89.958 76.436 62.994
52 100.000 100.000 90.075 76.671 63.313
53 100.000 100.000 90.521 77.449 64.223
54 100.000 100.000 90.756 77.948 64.900
55 100.000 100.000 90.943 78.312 65.377
56 100.000 100.000 91.389 79.090 66.287
57 100.000 100.000 91.436 79.197 66.452
58 100.000 100.000 91.600 79.524 66.890
59 100.000 100.000 91.835 79.969 67.458
60 100.000 100.000 92.046 80.355 67.935
61 100.000 100.000 92.140 80.569 68.251
62 100.000 100.000 92.210 80.714 68.452
63 100.000 100.000 92.351 81.020 68.887
64 100.000 100.000 92.398 81.127 69.052
65 100.000 100.000 92.445 81.250 69.253
66 100.000 100.000 92.445 81.283 69.336
67 100.000 100.000 92.586 81.577 69.747
68 100.000 100.000 92.633 81.690 69.924
69 100.000 100.000 92.679 81.803 70.102
70 100.000 100.000 92.726 81.910 70.265
71 100.000 100.000 92.773 82.012 70.414
72 100.000 100.000 92.820 82.114 70.564
73 100.000 100.000 92.938 82.371 70.934
74 100.000 100.000 92.985 82.489 71.125
75 100.000 100.000 93.055 82.660 71.393
76 100.000 100.000 93.196 82.955 71.788
77 100.000 100.000 93.407 83.374 72.334
78 100.000 100.000 93.430 83.438 72.445
79 100.000 100.000 93.618 83.813 72.942
80 100.000 100.000 93.712 84.006 73.210
81 100.000 100.000 93.993 84.553 73.917
82 100.000 100.000 94.040 84.687 74.142
83 100.000 100.000 94.064 84.757 74.268
84 100.000 100.000 94.087 84.821 74.379
85 100.000 100.000 94.158 84.966 74.581
86 100.000 100.000 94.181 85.030 74.692
87 100.000 100.000 94.369 85.373 75.117
88 100.000 100.000 94.369 85.406 75.200
89 100.000 100.000 94.510 85.690 75.587
90 100.000 100.000 94.627 85.920 75.894
91 100.000 100.000 94.674 86.070 76.160
92 100.000 100.000 94.721 86.172 76.310
93 100.000 100.000 94.862 86.456 76.696
94 100.000 100.000 94.932 86.633 76.970
95 100.000 100.000 95.026 86.826 77.238
96 100.000 100.000 95.096 86.977 77.454
97 100.000 100.000 95.214 87.245 77.850
98 100.000 100.000 95.354 87.550 78.285
99 100.000 100.000 95.354 87.582 78.372
100 100.000 100.000 95.378 87.641 78.470
101 100.000 100.000 95.495 87.883 78.801
102 100.000 100.000 95.589 88.086 79.096
103 100.000 100.000 95.659 88.231 79.298
104 100.000 100.000 95.706 88.333 79.448
105 100.000 100.000 95.753 88.446 79.625
106 100.000 100.000 95.894 88.719 79.984
107 100.000 100.000 96.176 89.255 80.655
108 100.000 100.000 96.269 89.448 80.927
109 100.000 100.000 96.340 89.604 81.156
110 100.000 100.000 96.340 89.636 81.247
111 100.000 100.000 96.340 89.652 81.293
112 100.000 100.000 96.387 89.754 81.443
113 100.000 100.000 96.481 89.958 81.735
114 100.000 100.000 96.481 89.968 81.769
115 100.000 100.000 96.527 90.076 81.933
116 100.000 100.000 96.692 90.397 82.355
117 100.000 100.000 96.762 90.569 82.621
118 100.000 100.000 96.786 90.622 82.704
119 100.000 100.000 96.809 90.676 82.788
120 100.000 100.000 96.809 90.698 82.847
121 100.000 100.000 96.856 90.799 82.996
122 100.000 100.000 96.903 90.901 83.146
123 100.000 100.000 96.950 91.014 83.324
124 100.000 100.000 96.997 91.121 83.488
125 100.000 100.000 97.067 91.293 83.756
126 100.000 100.000 97.114 91.400 83.921
127 100.000 100.000 97.114 91.421 83.984
128 100.000 100.000 97.137 91.507 84.145
129 100.000 100.000 97.161 91.598 84.319
130 100.000 100.000 97.161 91.609 84.350
131 100.000 100.000 97.161 91.641 84.433
132 100.000 100.000 97.255 91.850 84.731
133 100.000 100.000 97.255 91.893 84.838
134 100.000 100.000 97.302 91.995 84.988
135 100.000 100.000 97.325 92.054 85.086
136 100.000 100.000 97.396 92.204 85.302
137 100.000 100.000 97.396 92.210 85.319
138 100.000 100.000 97.396 92.285 85.502
139 100.000 100.000 97.396 92.295 85.534
140 100.000 100.000 97.419 92.376 85.687
141 100.000 100.000 97.419 92.392 85.731
142 100.000 100.000 97.513 92.596 86.024
143 100.000 100.000 97.536 92.660 86.136
144 100.000 100.000 97.583 92.762 86.286
145 100.000 100.000 97.607 92.815 86.369
146 100.000 100.000 97.630 92.869 86.453
147 100.000 100.000 97.654 92.923 86.536
148 100.000 100.000 97.654 92.923 86.540
149 100.000 100.000 97.654 92.928 86.556
150 100.000 100.000 97.701 93.030 86.706
151 100.000 100.000 97.748 93.142 86.884
152 100.000 100.000 97.771 93.207 86.998
153 100.000 100.000 97.818 93.330 87.200
154 100.000 100.000 97.818 93.346 87.246
155 100.000 100.000 97.888 93.496 87.462
156 100.000 100.000 97.888 93.496 87.466
157 100.000 100.000 97.912 93.577 87.616
158 100.000 100.000 97.912 93.582 87.632
159 100.000 100.000 97.912 93.614 87.721
160 100.000 100.000 97.912 93.641 87.797
161 100.000 100.000 97.912 93.646 87.814
162 100.000 100.000 97.959 93.748 87.963
163 100.000 100.000 97.959 93.759 87.995
164 100.000 100.000 98.006 93.861 88.145
165 100.000 100.000 98.029 93.920 88.243
166 100.000 100.000 98.053 93.995 88.383
167 100.000 100.000 98.099 94.102 88.552
168 100.000 100.000 98.123 94.172 88.681
169 100.000 100.000 98.123 94.193 88.740
170 100.000 100.000 98.170 94.295 88.890
171 100.000 100.000 98.170 94.376 89.090
172 100.000 100.000 98.193 94.429 89.173
173 100.000 100.000 98.287 94.633 89.465
174 100.000 100.000 98.358 94.778 89.667
175 100.000 100.000 98.498 95.073 90.077
176 100.000 100.000 98.592 95.260 90.332
177 100.000 100.000 98.592 95.266 90.349
178 100.000 100.000 98.639 95.368 90.499
179 100.000 100.000 98.639 95.373 90.515
180 100.000 100.000 98.686 95.528 90.789
181 100.000 100.000 98.733 95.630 90.939
182 100.000 100.000 98.756 95.684 91.022
183 100.000 100.000 98.780 95.737 91.105
184 100.000 100.000 98.780 95.754 91.155
185 100.000 100.000 98.803 95.813 91.253
186 100.000 100.000 98.827 95.866 91.336
187 100.000 100.000 98.850 95.952 91.499
188 100.000 100.000 98.850 95.963 91.531
189 100.000 100.000 98.874 96.032 91.658
190 100.000 100.000 98.874 96.048 91.704
191 100.000 100.000 98.921 96.150 91.856
192 100.000 100.000 98.921 96.161 91.887
193 100.000 100.000 98.921 96.172 91.919
194 100.000 100.000 98.921 96.193 91.980
195 100.000 100.000 98.921 96.220 92.056
196 100.000 100.000 98.944 96.274 92.139
197 100.000 100.000 98.944 96.290 92.185
198 100.000 100.000 98.968 96.349 92.283
199 100.000 100.000 99.015 96.451 92.433
200 100.000 100.000 99.015 96.472 92.492
201 100.000 100.000 99.061 96.574 92.642
202 100.000 100.000 99.085 96.628 92.725
203 100.000 100.000 99.108 96.681 92.808
204 100.000 100.000 99.108 96.729 92.932
205 100.000 100.000 99.108 96.740 92.964
206 100.000 100.000 99.108 96.756 93.008
207 100.000 100.000 99.108 96.767 93.040
208 100.000 100.000 99.108 96.772 93.056
209 100.000 100.000 99.108 96.778 93.073
210 100.000 100.000 99.108 96.794 93.119
211 100.000 100.000 99.132 96.847 93.204
212 100.000 100.000 99.155 96.901 93.287
213 100.000 100.000 99.155 96.906 93.306
214 100.000 100.000 99.155 96.928 93.365
215 100.000 100.000 99.155 96.933 93.384
216 100.000 100.000 99.179 96.992 93.482
217 100.000 100.000 99.179 96.997 93.498
218 100.000 100.000 99.226 97.121 93.700
219 100.000 100.000 99.226 97.121 93.707
220 100.000 100.000 99.226 97.126 93.724
221 100.000 100.000 99.320 97.341 94.044
222 100.000 100.000 99.320 97.357 94.088
223 100.000 100.000 99.343 97.410 94.171
224 100.000 100.000 99.343 97.416 94.188
225 100.000 100.000 99.343 97.432 94.238
226 100.000 100.000 99.343 97.432 94.242
227 100.000 100.000 99.390 97.560 94.460
228 100.000 100.000 99.413 97.614 94.543
229 100.000 100.000 99.460 97.721 94.706
230 100.000 100.000 99.460 97.732 94.737
231 100.000 100.000 99.460 97.737 94.756
232 100.000 100.000 99.460 97.743 94.772
233 100.000 100.000 99.554 97.946 95.068
234 100.000 100.000 99.554 97.963 95.114
235 100.000 100.000 99.578 98.016 95.198
236 100.000 100.000 99.578 98.038 95.257
237 100.000 100.000 99.578 98.043 95.273
238 100.000 100.000 99.578 98.043 95.275
239 100.000 100.000 99.578 98.064 95.334
240 100.000 100.000 99.578 98.086 95.394
241 100.000 100.000 99.578 98.091 95.410
242 100.000 100.000 99.578 98.097 95.427
243 100.000 100.000 99.578 98.107 95.458
244 100.000 100.000 99.601 98.161 95.542
245 100.000 100.000 99.625 98.215 95.625
246 100.000 100.000 99.625 98.215 95.627
247 100.000 100.000 99.672 98.327 95.804
248 100.000 100.000 99.695 98.397 95.926
249 100.000 100.000 99.695 98.408 95.961
250 100.000 100.000 99.695 98.418 95.993
251 100.000 100.000 99.695 98.429 96.024
252 100.000 100.000 99.695 98.450 96.087
253 100.000 100.000 99.742 98.552 96.237
254 100.000 100.000 99.742 98.558 96.253
255 100.000 100.000 99.742 98.568 96.289
256 100.000 100.000 99.742 98.601 96.376
257 100.000 100.000 99.742 98.606 96.392
258 100.000 100.000 99.765 98.660 96.475
259 100.000 100.000 99.765 98.665 96.492
260 100.000 100.000 99.812 98.778 96.670
261 100.000 100.000 99.812 98.788 96.701
262 100.000 100.000 99.836 98.853 96.812
263 100.000 100.000 99.836 98.858 96.829
264 100.000 100.000 99.836 98.858 96.836
265 100.000 100.000 99.859 98.912 96.919
266 100.000 100.000 99.859 98.917 96.936
267 100.000 100.000 99.859 98.944 97.010
268 100.000 100.000 99.859 98.944 97.012
269 100.000 100.000 99.859 98.954 97.043
270 100.000 100.000 99.859 98.965 97.075
271 100.000 100.000 99.906 99.067 97.224
272 100.000 100.000 99.930 99.121 97.308
273 100.000 100.000 99.930 99.185 97.470
274 100.000 100.000 99.930 99.185 97.474
275 100.000 100.000 99.930 99.190 97.491
276 100.000 100.000 99.930 99.196 97.507
277 100.000 100.000 99.930 99.217 97.568
278 100.000 100.000 99.930 99.223 97.585
279 100.000 100.000 99.930 99.233 97.616
280 100.000 100.000 99.930 99.239 97.633
281 100.000 100.000 99.930 99.239 97.635
282 100.000 100.000 99.930 99.244 97.653
283 100.000 100.000 99.930 99.260 97.698
284 100.000 100.000 99.930 99.287 97.770
285 100.000 100.000 99.930 99.308 97.829
286 100.000 100.000 99.930 99.319 97.860
287 100.000 100.000 99.930 99.324 97.877
288 100.000 100.000 99.930 99.335 97.909
289 100.000 100.000 99.930 99.335 97.910
290 100.000 100.000 99.930 99.341 97.927
291 100.000 100.000 99.930 99.351 97.962
292 100.000 100.000 99.930 99.351 97.964
293 100.000 100.000 99.930 99.362 97.995
294 100.000 100.000 99.930 99.367 98.012
295 100.000 100.000 99.930 99.367 98.014
296 100.000 100.000 99.930 99.373 98.031
297 100.000 100.000 99.930 99.378 98.047
298 100.000 100.000 99.930 99.383 98.064
299 100.000 100.000 99.930 99.389 98.081
300 100.000 100.000 99.930 99.394 98.097
301 100.000 100.000 99.930 99.399 98.119
302 100.000 100.000 99.930 99.410 98.154
303 100.000 100.000 99.930 99.421 98.186
304 100.000 100.000 99.930 99.442 98.245
305 100.000 100.000 99.930 99.464 98.308
306 100.000 100.000 99.930 99.469 98.325
307 100.000 100.000 99.930 99.485 98.371
308 100.000 100.000 99.930 99.496 98.402
309 100.000 100.000 99.953 99.550 98.485
310 100.000 100.000 99.953 99.555 98.502
311 100.000 100.000 99.953 99.560 98.519
312 100.000 100.000 99.953 99.566 98.535
313 100.000 100.000 99.953 99.566 98.537
314 100.000 100.000 99.953 99.571 98.554
315 100.000 100.000 99.953 99.582 98.589
316 100.000 100.000 100.000 99.694 98.767
317 100.000 100.000 100.000 99.716 98.826
318 100.000 100.000 100.000 99.727 98.857
319 100.000 100.000 100.000 99.732 98.874
320 100.000 100.000 100.000 99.743 98.905
321 100.000 100.000 100.000 99.748 98.922
322 100.000 100.000 100.000 99.748 98.931
323 100.000 100.000 100.000 99.748 98.935
324 100.000 100.000 100.000 99.753 98.951
325 100.000 100.000 100.000 99.759 98.968
326 100.000 100.000 100.000 99.759 98.970
327 100.000 100.000 100.000 99.759 98.977
328 100.000 100.000 100.000 99.759 98.981
329 100.000 100.000 100.000 99.769 99.016
330 100.000 100.000 100.000 99.780 99.048
331 100.000 100.000 100.000 99.786 99.064
332 100.000 100.000 100.000 99.791 99.081
333 100.000 100.000 100.000 99.812 99.140
334 100.000 100.000 100.000 99.823 99.172
335 100.000 100.000 100.000 99.828 99.188
336 100.000 100.000 100.000 99.834 99.205
337 100.000 100.000 100.000 99.845 99.236
338 100.000 100.000 100.000 99.866 99.295
339 100.000 100.000 100.000 99.866 99.299
340 100.000 100.000 100.000 99.866 99.301
341 100.000 100.000 100.000 99.877 99.336
342 100.000 100.000 100.000 99.877 99.338
343 100.000 100.000 100.000 99.887 99.369
344 100.000 100.000 100.000 99.893 99.386
345 100.000 100.000 100.000 99.903 99.417
346 100.000 100.000 100.000 99.914 99.449
347 100.000 100.000 100.000 99.920 99.466
348 100.000 100.000 100.000 99.920 99.469
349 100.000 100.000 100.000 99.930 99.504
350 100.000 100.000 100.000 99.936 99.521
351 100.000 100.000 100.000 99.941 99.538
352 100.000 100.000 100.000 99.946 99.554
353 100.000 100.000 100.000 99.957 99.586
354 100.000 100.000 100.000 99.973 99.630
355 100.000 100.000 100.000 99.979 99.647
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 61.905 33.841 19.052 11.318 7.116
4 71.429 41.794 24.472 14.857 9.438
5 78.571 48.900 29.634 18.321 11.741
6 85.714 55.330 34.374 21.613 13.991
7 88.095 59.391 38.128 24.530 16.112
8 92.857 63.959 41.835 27.334 18.159
9 97.619 68.866 45.730 30.197 20.214
10 100.000 72.420 49.061 32.867 22.215
11 100.000 74.958 51.971 35.355 24.140
12 100.000 77.157 54.622 37.708 26.009
13 100.000 79.188 57.156 39.998 27.844
14 100.000 81.049 59.456 42.110 29.582
15 100.000 82.572 61.520 44.116 31.283
16 100.000 84.095 63.491 45.987 32.864
17 100.000 85.279 65.275 47.820 34.475
18 100.000 86.464 66.917 49.467 35.927
19 100.000 87.479 68.466 51.102 37.408
20 100.000 88.663 70.108 52.748 38.859
21 100.000 89.679 71.563 54.254 40.228
22 100.000 90.694 72.994 55.718 41.548
23 100.000 91.371 74.214 57.128 42.896
24 100.000 92.047 75.340 58.394 44.104
25 100.000 92.724 76.420 59.605 45.276
26 100.000 93.401 77.475 60.790 46.430
27 100.000 93.739 78.297 61.879 47.567
28 100.000 94.416 79.282 62.930 48.562
29 100.000 94.585 79.962 63.932 49.653
30 100.000 95.093 80.831 64.951 50.667
31 100.000 95.601 81.699 65.964 51.674
32 100.000 96.108 82.543 66.962 52.688
33 100.000 96.616 83.365 67.894 53.607
34 100.000 96.954 84.069 68.801 54.563
35 100.000 97.124 84.679 69.696 55.537
36 100.000 97.631 85.453 70.565 56.401
37 100.000 97.970 86.133 71.428 57.309
38 100.000 98.308 86.767 72.205 58.112
39 100.000 98.646 87.400 72.988 58.927
40 100.000 98.816 87.870 73.701 59.767
41 100.000 98.985 88.339 74.377 60.529
42 100.000 99.323 88.855 74.956 61.107
43 100.000 99.323 89.183 75.551 61.832
44 100.000 99.492 89.606 76.157 62.528
45 100.000 99.492 89.934 76.752 63.252
46 100.000 99.492 90.239 77.320 63.959
47 100.000 99.492 90.544 77.883 64.656
48 100.000 99.492 90.826 78.414 65.325
49 100.000 99.492 91.084 78.907 65.956
50 100.000 99.492 91.342 79.406 66.599
51 100.000 99.492 91.577 79.862 67.191
52 100.000 99.492 91.811 80.307 67.759
53 100.000 99.662 92.140 80.746 68.255
54 100.000 99.831 92.468 81.207 68.802
55 100.000 99.831 92.679 81.626 69.347
56 100.000 99.831 92.891 82.028 69.863
57 100.000 99.831 93.102 82.441 70.403
58 100.000 99.831 93.313 82.859 70.956
59 100.000 99.831 93.501 83.213 71.406
60 100.000 99.831 93.688 83.583 71.894
61 100.000 100.000 93.970 83.926 72.265
62 100.000 100.000 94.158 84.296 72.756
63 100.000 100.000 94.345 84.671 73.257
64 100.000 100.000 94.533 85.025 73.706
65 100.000 100.000 94.697 85.352 74.144
66 100.000 100.000 94.862 85.700 74.631
67 100.000 100.000 95.002 85.985 75.010
68 100.000 100.000 95.143 86.290 75.444
69 100.000 100.000 95.284 86.585 75.857
70 100.000 100.000 95.425 86.875 76.258
71 100.000 100.000 95.565 87.148 76.617
72 100.000 100.000 95.683 87.389 76.953
73 100.000 100.000 95.800 87.636 77.301
74 100.000 100.000 95.917 87.883 77.649
75 100.000 100.000 96.035 88.129 77.996
76 100.000 100.000 96.152 88.387 78.368
77 100.000 100.000 96.269 88.623 78.688
78 100.000 100.000 96.387 88.864 79.023
79 100.000 100.000 96.481 89.057 79.291
80 100.000 100.000 96.574 89.271 79.614
81 100.000 100.000 96.668 89.480 79.919
82 100.000 100.000 96.762 89.673 80.188
83 100.000 100.000 96.856 89.888 80.507
84 100.000 100.000 96.950 90.092 80.796
85 100.000 100.000 97.044 90.285 81.064
86 100.000 100.000 97.137 90.478 81.332
87 100.000 100.000 97.231 90.703 81.678
88 100.000 100.000 97.325 90.917 81.998
89 100.000 100.000 97.419 91.110 82.266
90 100.000 100.000 97.489 91.261 82.482
91 100.000 100.000 97.560 91.432 82.754
92 100.000 100.000 97.630 91.588 82.984
93 100.000 100.000 97.701 91.732 83.185
94 100.000 100.000 97.771 91.893 83.429
95 100.000 100.000 97.841 92.054 83.671
96 100.000 100.000 97.912 92.204 83.888
97 100.000 100.000 97.982 92.392 84.193
98 100.000 100.000 98.053 92.547 84.422
99 100.000 100.000 98.123 92.697 84.639
100 100.000 100.000 98.193 92.848 84.855
101 100.000 100.000 98.264 93.030 85.147
102 100.000 100.000 98.334 93.207 85.428
103 100.000 100.000 98.381 93.319 85.606
104 100.000 100.000 98.428 93.432 85.783
105 100.000 100.000 98.475 93.561 85.998
106 100.000 100.000 98.522 93.684 86.201
107 100.000 100.000 98.569 93.813 86.419
108 100.000 100.000 98.616 93.915 86.569
109 100.000 100.000 98.663 94.038 86.771
110 100.000 100.000 98.710 94.167 86.989
111 100.000 100.000 98.756 94.279 87.168
112 100.000 100.000 98.803 94.402 87.370
113 100.000 100.000 98.850 94.504 87.520
114 100.000 100.000 98.897 94.606 87.669
115 100.000 100.000 98.944 94.708 87.819
116 100.000 100.000 98.991 94.831 88.021
117 100.000 100.000 99.038 94.933 88.171
118 100.000 100.000 99.085 95.046 88.350
119 100.000 100.000 99.132 95.158 88.527
120 100.000 100.000 99.179 95.271 88.705
121 100.000 100.000 99.226 95.373 88.855
122 100.000 100.000 99.249 95.427 88.938
123 100.000 100.000 99.273 95.485 89.036
124 100.000 100.000 99.296 95.550 89.149
125 100.000 100.000 99.320 95.609 89.247
126 100.000 100.000 99.343 95.679 89.373
127 100.000 100.000 99.366 95.743 89.485
128 100.000 100.000 99.390 95.813 89.611
129 100.000 100.000 99.413 95.866 89.694
130 100.000 100.000 99.437 95.941 89.833
131 100.000 100.000 99.460 96.038 90.025
132 100.000 100.000 99.484 96.113 90.164
133 100.000 100.000 99.507 96.172 90.262
134 100.000 100.000 99.531 96.225 90.345
135 100.000 100.000 99.554 96.290 90.456
136 100.000 100.000 99.578 96.365 90.595
137 100.000 100.000 99.601 96.445 90.747
138 100.000 100.000 99.625 96.510 90.857
139 100.000 100.000 99.648 96.585 90.996
140 100.000 100.000 99.672 96.638 91.079
141 100.000 100.000 99.695 96.708 91.205
142 100.000 100.000 99.718 96.767 91.303
143 100.000 100.000 99.742 96.821 91.386
144 100.000 100.000 99.765 96.906 91.549
145 100.000 100.000 99.789 96.976 91.673
146 100.000 100.000 99.812 97.030 91.756
147 100.000 100.000 99.836 97.089 91.858
148 100.000 100.000 99.859 97.142 91.941
149 100.000 100.000 99.883 97.201 92.039
150 100.000 100.000 99.906 97.266 92.150
151 100.000 100.000 99.930 97.325 92.248
152 100.000 100.000 99.953 97.378 92.331
153 100.000 100.000 99.977 97.432 92.415
154 100.000 100.000 100.000 97.501 92.540
155 100.000 100.000 100.000 97.512 92.572
156 100.000 100.000 100.000 97.518 92.588
157 100.000 100.000 100.000 97.534 92.633
158 100.000 100.000 100.000 97.544 92.666
159 100.000 100.000 100.000 97.560 92.712
160 100.000 100.000 100.000 97.566 92.731
161 100.000 100.000 100.000 97.593 92.805
162 100.000 100.000 100.000 97.614 92.864
163 100.000 100.000 100.000 97.630 92.910
164 100.000 100.000 100.000 97.668 93.006
165 100.000 100.000 100.000 97.700 93.093
166 100.000 100.000 100.000 97.716 93.138
167 100.000 100.000 100.000 97.727 93.169
168 100.000 100.000 100.000 97.748 93.228
169 100.000 100.000 100.000 97.759 93.260
170 100.000 100.000 100.000 97.775 93.306
171 100.000 100.000 100.000 97.780 93.326
172 100.000 100.000 100.000 97.786 93.345
173 100.000 100.000 100.000 97.807 93.404
174 100.000 100.000 100.000 97.812 93.420
175 100.000 100.000 100.000 97.823 93.452
176 100.000 100.000 100.000 97.845 93.511
177 100.000 100.000 100.000 97.855 93.543
178 100.000 100.000 100.000 97.877 93.605
179 100.000 100.000 100.000 97.946 93.777
180 100.000 100.000 100.000 97.963 93.822
181 100.000 100.000 100.000 97.979 93.866
182 100.000 100.000 100.000 97.984 93.885
183 100.000 100.000 100.000 98.005 93.944
184 100.000 100.000 100.000 98.016 93.975
185 100.000 100.000 100.000 98.027 94.010
186 100.000 100.000 100.000 98.059 94.094
187 100.000 100.000 100.000 98.075 94.140
188 100.000 100.000 100.000 98.102 94.212
189 100.000 100.000 100.000 98.113 94.243
190 100.000 100.000 100.000 98.123 94.279
191 100.000 100.000 100.000 98.198 94.458
192 100.000 100.000 100.000 98.204 94.475
193 100.000 100.000 100.000 98.247 94.585
194 100.000 100.000 100.000 98.257 94.621
195 100.000 100.000 100.000 98.322 94.780
196 100.000 100.000 100.000 98.333 94.811
197 100.000 100.000 100.000 98.365 94.898
198 100.000 100.000 100.000 98.370 94.915
199 100.000 100.000 100.000 98.386 94.961
200 100.000 100.000 100.000 98.402 95.007
201 100.000 100.000 100.000 98.429 95.079
202 100.000 100.000 100.000 98.445 95.125
203 100.000 100.000 100.000 98.467 95.185
204 100.000 100.000 100.000 98.483 95.233
205 100.000 100.000 100.000 98.499 95.277
206 100.000 100.000 100.000 98.515 95.325
207 100.000 100.000 100.000 98.531 95.370
208 100.000 100.000 100.000 98.542 95.401
209 100.000 100.000 100.000 98.552 95.447
210 100.000 100.000 100.000 98.568 95.493
211 100.000 100.000 100.000 98.601 95.577
212 100.000 100.000 100.000 98.606 95.593
213 100.000 100.000 100.000 98.649 95.704
214 100.000 100.000 100.000 98.660 95.736
215 100.000 100.000 100.000 98.665 95.754
216 100.000 100.000 100.000 98.670 95.771
217 100.000 100.000 100.000 98.676 95.787
218 100.000 100.000 100.000 98.697 95.847
219 100.000 100.000 100.000 98.713 95.891
220 100.000 100.000 100.000 98.719 95.908
221 100.000 100.000 100.000 98.724 95.924
222 100.000 100.000 100.000 98.745 95.984
223 100.000 100.000 100.000 98.751 96.000
224 100.000 100.000 100.000 98.756 96.017
225 100.000 100.000 100.000 98.761 96.033
226 100.000 100.000 100.000 98.778 96.080
227 100.000 100.000 100.000 98.778 96.089
228 100.000 100.000 100.000 98.788 96.120
229 100.000 100.000 100.000 98.831 96.231
230 100.000 100.000 100.000 98.837 96.248
231 100.000 100.000 100.000 98.842 96.266
232 100.000 100.000 100.000 98.853 96.298
233 100.000 100.000 100.000 98.879 96.370
234 100.000 100.000 100.000 98.912 96.457
235 100.000 100.000 100.000 98.917 96.474
236 100.000 100.000 100.000 98.922 96.490
237 100.000 100.000 100.000 98.944 96.549
238 100.000 100.000 100.000 98.949 96.568
239 100.000 100.000 100.000 98.971 96.627
240 100.000 100.000 100.000 99.003 96.714
241 100.000 100.000 100.000 99.013 96.745
242 100.000 100.000 100.000 99.019 96.762
243 100.000 100.000 100.000 99.035 96.808
244 100.000 100.000 100.000 99.040 96.825
245 100.000 100.000 100.000 99.056 96.869
246 100.000 100.000 100.000 99.056 96.873
247 100.000 100.000 100.000 99.078 96.932
248 100.000 100.000 100.000 99.089 96.967
249 100.000 100.000 100.000 99.094 96.986
250 100.000 100.000 100.000 99.105 97.017
251 100.000 100.000 100.000 99.121 97.065
252 100.000 100.000 100.000 99.131 97.097
253 100.000 100.000 100.000 99.137 97.115
254 100.000 100.000 100.000 99.147 97.150
255 100.000 100.000 100.000 99.164 97.197
256 100.000 100.000 100.000 99.169 97.213
257 100.000 100.000 100.000 99.180 97.245
258 100.000 100.000 100.000 99.190 97.276
259 100.000 100.000 100.000 99.212 97.339
260 100.000 100.000 100.000 99.217 97.356
261 100.000 100.000 100.000 99.282 97.518
262 100.000 100.000 100.000 99.292 97.550
263 100.000 100.000 100.000 99.303 97.581
264 100.000 100.000 100.000 99.303 97.585
265 100.000 100.000 100.000 99.330 97.661
266 100.000 100.000 100.000 99.330 97.664
267 100.000 100.000 100.000 99.335 97.681
268 100.000 100.000 100.000 99.341 97.701
269 100.000 100.000 100.000 99.351 97.733
270 100.000 100.000 100.000 99.357 97.750
271 100.000 100.000 100.000 99.378 97.809
272 100.000 100.000 100.000 99.389 97.840
273 100.000 100.000 100.000 99.394 97.857
274 100.000 100.000 100.000 99.405 97.888
275 100.000 100.000 100.000 99.410 97.905
276 100.000 100.000 100.000 99.416 97.923
277 100.000 100.000 100.000 99.421 97.942
278 100.000 100.000 100.000 99.426 97.958
279 100.000 100.000 100.000 99.432 97.975
280 100.000 100.000 100.000 99.442 98.007
281 100.000 100.000 100.000 99.448 98.023
282 100.000 100.000 100.000 99.469 98.082
283 100.000 100.000 100.000 99.501 98.169
284 100.000 100.000 100.000 99.501 98.173
285 100.000 100.000 100.000 99.501 98.175
286 100.000 100.000 100.000 99.512 98.206
287 100.000 100.000 100.000 99.534 98.265
288 100.000 100.000 100.000 99.544 98.297
289 100.000 100.000 100.000 99.555 98.328
290 100.000 100.000 100.000 99.555 98.330
291 100.000 100.000 100.000 99.566 98.362
292 100.000 100.000 100.000 99.571 98.378
293 100.000 100.000 100.000 99.576 98.395
294 100.000 100.000 100.000 99.587 98.426
295 100.000 100.000 100.000 99.593 98.443
296 100.000 100.000 100.000 99.598 98.460
297 100.000 100.000 100.000 99.598 98.463
298 100.000 100.000 100.000 99.609 98.495
299 100.000 100.000 100.000 99.609 98.498
300 100.000 100.000 100.000 99.619 98.530
301 100.000 100.000 100.000 99.625 98.547
302 100.000 100.000 100.000 99.635 98.582
303 100.000 100.000 100.000 99.646 98.613
304 100.000 100.000 100.000 99.651 98.630
305 100.000 100.000 100.000 99.662 98.661
306 100.000 100.000 100.000 99.668 98.678
307 100.000 100.000 100.000 99.668 98.685
308 100.000 100.000 100.000 99.700 98.772
309 100.000 100.000 100.000 99.721 98.831
310 100.000 100.000 100.000 99.727 98.848
311 100.000 100.000 100.000 99.732 98.866
312 100.000 100.000 100.000 99.753 98.926
313 100.000 100.000 100.000 99.764 98.957
314 100.000 100.000 100.000 99.769 98.974
315 100.000 100.000 100.000 99.780 99.005
316 100.000 100.000 100.000 99.786 99.022
317 100.000 100.000 100.000 99.818 99.109
318 100.000 100.000 100.000 99.823 99.125
319 100.000 100.000 100.000 99.823 99.129
320 100.000 100.000 100.000 99.823 99.133
321 100.000 100.000 100.000 99.823 99.135
322 100.000 100.000 100.000 99.828 99.151
323 100.000 100.000 100.000 99.828 99.153
324 100.000 100.000 100.000 99.828 99.160
325 100.000 100.000 100.000 99.828 99.162
326 100.000 100.000 100.000 99.828 99.164
327 100.000 100.000 100.000 99.839 99.196
328 100.000 100.000 100.000 99.861 99.255
329 100.000 100.000 100.000 99.861 99.257
330 100.000 100.000 100.000 99.866 99.273
331 100.000 100.000 100.000 99.871 99.295
332 100.000 100.000 100.000 99.877 99.312
333 100.000 100.000 100.000 99.887 99.344
334 100.000 100.000 100.000 99.887 99.345
335 100.000 100.000 100.000 99.898 99.377
336 100.000 100.000 100.000 99.909 99.408
337 100.000 100.000 100.000 99.920 99.440
338 100.000 100.000 100.000 99.925 99.456
339 100.000 100.000 100.000 99.925 99.458
340 100.000 100.000 100.000 99.930 99.475
341 100.000 100.000 100.000 99.941 99.510
342 100.000 100.000 100.000 99.946 99.527
343 100.000 100.000 100.000 99.946 99.530
344 100.000 100.000 100.000 99.946 99.532
345 100.000 100.000 100.000 99.952 99.549
346 100.000 100.000 100.000 99.962 99.580
347 100.000 100.000 100.000 99.968 99.597
348 100.000 100.000 100.000 99.973 99.614
349 100.000 100.000 100.000 99.979 99.630
350 100.000 100.000 100.000 99.984 99.647
351 100.000 100.000 100.000 99.984 99.650
352 100.000 100.000 100.000 99.984 99.654
353 100.000 100.000 100.000 99.989 99.671
354 100.000 100.000 100.000 99.989 99.673
355 100.000 100.000 100.000 99.995 99.689
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 45.238 22.843 12.600 7.479 4.721
3 59.524 32.826 18.700 11.211 7.088
4 69.048 40.778 24.097 14.728 9.400
5 76.190 47.547 29.071 18.128 11.687
6 80.952 53.299 33.693 21.420 13.947
7 83.333 57.868 37.799 24.503 16.131
8 88.095 62.944 41.952 27.575 18.309
9 90.476 66.328 45.096 30.122 20.251
10 95.238 70.897 48.733 32.851 22.250
11 97.619 74.112 51.807 35.387 24.195
12 97.619 75.973 54.223 37.633 26.028
13 100.000 79.019 57.039 39.955 27.842
14 100.000 80.541 59.057 41.896 29.489
15 100.000 81.387 60.535 43.553 31.022
16 100.000 82.741 62.459 45.456 32.653
17 100.000 84.095 64.313 47.284 34.236
18 100.000 84.772 65.580 48.753 35.634
19 100.000 85.787 67.081 50.319 37.049
20 100.000 86.633 68.419 51.772 38.399
21 100.000 87.310 69.568 53.107 39.703
22 100.000 88.494 71.164 54.683 41.088
23 100.000 89.340 72.478 56.104 42.410
24 100.000 90.017 73.627 57.407 43.656
25 100.000 90.186 74.402 58.522 44.829
26 100.000 90.863 75.504 59.756 46.008
27 100.000 91.709 76.725 61.037 47.198
28 100.000 92.386 77.757 62.168 48.287
29 100.000 92.555 78.390 63.096 49.287
30 100.000 93.232 79.399 64.200 50.356
31 100.000 93.570 80.103 65.101 51.297
32 100.000 93.909 80.784 65.975 52.226
33 100.000 94.585 81.746 66.994 53.193
34 100.000 95.093 82.543 67.916 54.128
35 100.000 95.431 83.200 68.752 55.020
36 100.000 95.601 83.717 69.508 55.872
37 100.000 95.939 84.374 70.334 56.738
38 100.000 96.616 85.312 71.304 57.642
39 100.000 96.954 85.946 72.103 58.489
40 100.000 97.293 86.579 72.902 59.339
41 100.000 97.293 86.884 73.497 60.101
42 100.000 97.293 87.189 74.066 60.811
43 100.000 97.800 87.893 74.838 61.590
44 100.000 97.800 88.198 75.401 62.287
45 100.000 97.800 88.480 75.937 62.958
46 100.000 97.970 88.878 76.521 63.646
47 100.000 97.970 89.160 77.047 64.305
48 100.000 98.477 89.887 77.814 65.026
49 100.000 98.816 90.427 78.446 65.684
50 100.000 98.816 90.662 78.902 66.276
51 100.000 98.985 91.037 79.433 66.894
52 100.000 98.985 91.272 79.888 67.485
53 100.000 99.154 91.647 80.409 68.079
54 100.000 99.154 91.811 80.784 68.621
55 100.000 99.154 91.999 81.181 69.166
56 100.000 99.154 92.210 81.599 69.712
57 100.000 99.323 92.539 82.065 70.272
58 100.000 99.492 92.891 82.553 70.838
59 100.000 99.492 93.078 82.939 71.363
60 100.000 99.492 93.290 83.341 71.879
61 100.000 99.492 93.477 83.717 72.380
62 100.000 99.492 93.665 84.092 72.881
63 100.000 99.492 93.853 84.462 73.371
64 100.000 99.492 94.017 84.805 73.847
65 100.000 99.662 94.345 85.245 74.342
66 100.000 99.831 94.650 85.647 74.804
67 100.000 99.831 94.791 85.947 75.228
68 100.000 99.831 94.932 86.247 75.651
69 100.000 99.831 95.073 86.548 76.075
70 100.000 99.831 95.214 86.843 76.485
71 100.000 99.831 95.307 87.089 76.874
72 100.000 99.831 95.425 87.352 77.260
73 100.000 99.831 95.542 87.609 77.632
74 100.000 99.831 95.636 87.845 77.996
75 100.000 99.831 95.753 88.103 78.368
76 100.000 99.831 95.894 88.381 78.741
77 100.000 99.831 96.011 88.633 79.102
78 100.000 99.831 96.105 88.859 79.446
79 100.000 99.831 96.222 89.105 79.794
80 100.000 99.831 96.340 89.352 80.141
81 100.000 99.831 96.457 89.598 80.489
82 100.000 99.831 96.527 89.797 80.818
83 100.000 99.831 96.621 90.017 81.151
84 100.000 99.831 96.715 90.226 81.460
85 100.000 99.831 96.809 90.429 81.748
86 100.000 99.831 96.879 90.612 82.044
87 100.000 99.831 96.973 90.810 82.329
88 100.000 99.831 97.067 91.009 82.608
89 100.000 99.831 97.114 91.164 82.886
90 100.000 100.000 97.372 91.459 83.191
91 100.000 100.000 97.466 91.657 83.477
92 100.000 100.000 97.560 91.850 83.745
93 100.000 100.000 97.607 92.000 84.010
94 100.000 100.000 97.677 92.172 84.280
95 100.000 100.000 97.748 92.338 84.533
96 100.000 100.000 97.818 92.504 84.790
97 100.000 100.000 97.888 92.671 85.047
98 100.000 100.000 97.959 92.831 85.288
99 100.000 100.000 98.029 92.992 85.534
100 100.000 100.000 98.099 93.148 85.763
101 100.000 100.000 98.170 93.298 85.979
102 100.000 100.000 98.217 93.427 86.197
103 100.000 100.000 98.287 93.577 86.414
104 100.000 100.000 98.358 93.727 86.630
105 100.000 100.000 98.428 93.877 86.847
106 100.000 100.000 98.498 94.027 87.063
107 100.000 100.000 98.569 94.177 87.279
108 100.000 100.000 98.639 94.327 87.496
109 100.000 100.000 98.686 94.451 87.701
110 100.000 100.000 98.733 94.574 87.906
111 100.000 100.000 98.780 94.697 88.111
112 100.000 100.000 98.827 94.821 88.319
113 100.000 100.000 98.874 94.944 88.520
114 100.000 100.000 98.921 95.057 88.705
115 100.000 100.000 98.968 95.169 88.883
116 100.000 100.000 99.015 95.282 89.060
117 100.000 100.000 99.061 95.389 89.225
118 100.000 100.000 99.061 95.453 89.387
119 100.000 100.000 99.108 95.561 89.552
120 100.000 100.000 99.155 95.668 89.717
121 100.000 100.000 99.179 95.754 89.887
122 100.000 100.000 99.226 95.861 90.055
123 100.000 100.000 99.249 95.947 90.221
124 100.000 100.000 99.249 96.011 90.377
125 100.000 100.000 99.273 96.091 90.532
126 100.000 100.000 99.320 96.193 90.682
127 100.000 100.000 99.343 96.274 90.837
128 100.000 100.000 99.366 96.354 90.992
129 100.000 100.000 99.413 96.456 91.142
130 100.000 100.000 99.437 96.536 91.296
131 100.000 100.000 99.484 96.638 91.446
132 100.000 100.000 99.484 96.692 91.581
133 100.000 100.000 99.507 96.767 91.719
134 100.000 100.000 99.531 96.842 91.860
135 100.000 100.000 99.554 96.917 92.000
136 100.000 100.000 99.578 96.987 92.126
137 100.000 100.000 99.601 97.056 92.252
138 100.000 100.000 99.625 97.126 92.378
139 100.000 100.000 99.648 97.190 92.488
140 100.000 100.000 99.672 97.255 92.599
141 100.000 100.000 99.695 97.319 92.712
142 100.000 100.000 99.718 97.384 92.825
143 100.000 100.000 99.742 97.448 92.938
144 100.000 100.000 99.765 97.512 93.052
145 100.000 100.000 99.789 97.577 93.163
146 100.000 100.000 99.789 97.614 93.263
147 100.000 100.000 99.812 97.673 93.361
148 100.000 100.000 99.836 97.732 93.459
149 100.000 100.000 99.859 97.791 93.557
150 100.000 100.000 99.883 97.850 93.655
151 100.000 100.000 99.883 97.888 93.752
152 100.000 100.000 99.883 97.925 93.851
153 100.000 100.000 99.883 97.963 93.955
154 100.000 100.000 99.906 98.022 94.057
155 100.000 100.000 99.906 98.054 94.144
156 100.000 100.000 99.906 98.086 94.230
157 100.000 100.000 99.930 98.140 94.314
158 100.000 100.000 99.953 98.193 94.397
159 100.000 100.000 99.953 98.225 94.484
160 100.000 100.000 99.977 98.279 94.567
161 100.000 100.000 99.977 98.311 94.654
162 100.000 100.000 99.977 98.343 94.745
163 100.000 100.000 99.977 98.375 94.835
164 100.000 100.000 100.000 98.429 94.918
165 100.000 100.000 100.000 98.456 94.990
166 100.000 100.000 100.000 98.483 95.063
167 100.000 100.000 100.000 98.509 95.135
168 100.000 100.000 100.000 98.536 95.211
169 100.000 100.000 100.000 98.563 95.283
170 100.000 100.000 100.000 98.590 95.355
171 100.000 100.000 100.000 98.617 95.427
172 100.000 100.000 100.000 98.638 95.490
173 100.000 100.000 100.000 98.660 95.549
174 100.000 100.000 100.000 98.681 95.608
175 100.000 100.000 100.000 98.702 95.667
176 100.000 100.000 100.000 98.724 95.728
177 100.000 100.000 100.000 98.745 95.791
178 100.000 100.000 100.000 98.767 95.850
179 100.000 100.000 100.000 98.788 95.911
180 100.000 100.000 100.000 98.810 95.971
181 100.000 100.000 100.000 98.831 96.033
182 100.000 100.000 100.000 98.853 96.093
183 100.000 100.000 100.000 98.874 96.152
184 100.000 100.000 100.000 98.896 96.211
185 100.000 100.000 100.000 98.917 96.274
186 100.000 100.000 100.000 98.938 96.335
187 100.000 100.000 100.000 98.960 96.394
188 100.000 100.000 100.000 98.981 96.453
189 100.000 100.000 100.000 99.003 96.512
190 100.000 100.000 100.000 99.024 96.572
191 100.000 100.000 100.000 99.046 96.631
192 100.000 100.000 100.000 99.067 96.690
193 100.000 100.000 100.000 99.089 96.751
194 100.000 100.000 100.000 99.105 96.797
195 100.000 100.000 100.000 99.121 96.843
196 100.000 100.000 100.000 99.137 96.893
197 100.000 100.000 100.000 99.153 96.945
198 100.000 100.000 100.000 99.169 96.991
199 100.000 100.000 100.000 99.185 97.038
200 100.000 100.000 100.000 99.201 97.086
201 100.000 100.000 100.000 99.217 97.132
202 100.000 100.000 100.000 99.233 97.178
203 100.000 100.000 100.000 99.249 97.224
204 100.000 100.000 100.000 99.265 97.271
205 100.000 100.000 100.000 99.282 97.317
206 100.000 100.000 100.000 99.298 97.365
207 100.000 100.000 100.000 99.314 97.409
208 100.000 100.000 100.000 99.324 97.441
209 100.000 100.000 100.000 99.335 97.472
210 100.000 100.000 100.000 99.346 97.504
211 100.000 100.000 100.000 99.357 97.535
212 100.000 100.000 100.000 99.367 97.566
213 100.000 100.000 100.000 99.378 97.598
214 100.000 100.000 100.000 99.389 97.629
215 100.000 100.000 100.000 99.399 97.661
216 100.000 100.000 100.000 99.410 97.694
217 100.000 100.000 100.000 99.421 97.725
218 100.000 100.000 100.000 99.432 97.761
219 100.000 100.000 100.000 99.442 97.792
220 100.000 100.000 100.000 99.453 97.823
221 100.000 100.000 100.000 99.464 97.855
222 100.000 100.000 100.000 99.475 97.886
223 100.000 100.000 100.000 99.485 97.918
224 100.000 100.000 100.000 99.496 97.949
225 100.000 100.000 100.000 99.507 97.981
226 100.000 100.000 100.000 99.517 98.012
227 100.000 100.000 100.000 99.528 98.044
228 100.000 100.000 100.000 99.539 98.075
229 100.000 100.000 100.000 99.550 98.106
230 100.000 100.000 100.000 99.560 98.138
231 100.000 100.000 100.000 99.571 98.171
232 100.000 100.000 100.000 99.582 98.204
233 100.000 100.000 100.000 99.593 98.236
234 100.000 100.000 100.000 99.603 98.267
235 100.000 100.000 100.000 99.614 98.299
236 100.000 100.000 100.000 99.625 98.334
237 100.000 100.000 100.000 99.635 98.365
238 100.000 100.000 100.000 99.646 98.397
239 100.000 100.000 100.000 99.657 98.428
240 100.000 100.000 100.000 99.668 98.463
241 100.000 100.000 100.000 99.678 98.495
242 100.000 100.000 100.000 99.684 98.513
243 100.000 100.000 100.000 99.689 98.530
244 100.000 100.000 100.000 99.694 98.558
245 100.000 100.000 100.000 99.700 98.574
246 100.000 100.000 100.000 99.705 98.591
247 100.000 100.000 100.000 99.710 98.608
248 100.000 100.000 100.000 99.716 98.624
249 100.000 100.000 100.000 99.721 98.641
250 100.000 100.000 100.000 99.727 98.657
251 100.000 100.000 100.000 99.732 98.676
252 100.000 100.000 100.000 99.737 98.694
253 100.000 100.000 100.000 99.743 98.711
254 100.000 100.000 100.000 99.748 98.730
255 100.000 100.000 100.000 99.753 98.746
256 100.000 100.000 100.000 99.759 98.763
257 100.000 100.000 100.000 99.764 98.780
258 100.000 100.000 100.000 99.769 98.796
259 100.000 100.000 100.000 99.775 98.813
260 100.000 100.000 100.000 99.780 98.831
261 100.000 100.000 100.000 99.786 98.848
262 100.000 100.000 100.000 99.791 98.865
263 100.000 100.000 100.000 99.796 98.881
264 100.000 100.000 100.000 99.802 98.898
265 100.000 100.000 100.000 99.807 98.915
266 100.000 100.000 100.000 99.812 98.931
267 100.000 100.000 100.000 99.818 98.948
268 100.000 100.000 100.000 99.823 98.964
269 100.000 100.000 100.000 99.828 98.981
270 100.000 100.000 100.000 99.834 98.998
271 100.000 100.000 100.000 99.839 99.014
272 100.000 100.000 100.000 99.845 99.031
273 100.000 100.000 100.000 99.850 99.050
274 100.000 100.000 100.000 99.855 99.066
275 100.000 100.000 100.000 99.861 99.083
276 100.000 100.000 100.000 99.866 99.099
277 100.000 100.000 100.000 99.871 99.116
278 100.000 100.000 100.000 99.877 99.133
279 100.000 100.000 100.000 99.882 99.151
280 100.000 100.000 100.000 99.887 99.168
281 100.000 100.000 100.000 99.893 99.184
282 100.000 100.000 100.000 99.898 99.201
283 100.000 100.000 100.000 99.903 99.221
284 100.000 100.000 100.000 99.909 99.240
285 100.000 100.000 100.000 99.914 99.257
286 100.000 100.000 100.000 99.920 99.275
287 100.000 100.000 100.000 99.925 99.292
288 100.000 100.000 100.000 99.930 99.308
289 100.000 100.000 100.000 99.936 99.325
290 100.000 100.000 100.000 99.941 99.342
291 100.000 100.000 100.000 99.946 99.358
292 100.000 100.000 100.000 99.952 99.377
293 100.000 100.000 100.000 99.957 99.393
294 100.000 100.000 100.000 99.962 99.410
295 100.000 100.000 100.000 99.968 99.429
296 100.000 100.000 100.000 99.973 99.447
297 100.000 100.000 100.000 99.979 99.464
298 100.000 100.000 100.000 99.984 99.480
299 100.000 100.000 100.000 99.989 99.497
300 100.000 100.000 100.000 99.995 99.514
301 100.000 100.000 100.000 100.000 99.530
302 100.000 100.000 100.000 100.000 99.532
303 100.000 100.000 100.000 100.000 99.540
304 100.000 100.000 100.000 100.000 99.543
305 100.000 100.000 100.000 100.000 99.545
306 100.000 100.000 100.000 100.000 99.549
307 100.000 100.000 100.000 100.000 99.552
308 100.000 100.000 100.000 100.000 99.554
309 100.000 100.000 100.000 100.000 99.556
310 100.000 100.000 100.000 100.000 99.560
311 100.000 100.000 100.000 100.000 99.564
312 100.000 100.000 100.000 100.000 99.565
313 100.000 100.000 100.000 100.000 99.569
314 100.000 100.000 100.000 100.000 99.573
315 100.000 100.000 100.000 100.000 99.577
316 100.000 100.000 100.000 100.000 99.580
317 100.000 100.000 100.000 100.000 99.584
318 100.000 100.000 100.000 100.000 99.588
319 100.000 100.000 100.000 100.000 99.591
320 100.000 100.000 100.000 100.000 99.599
321 100.000 100.000 100.000 100.000 99.601
322 100.000 100.000 100.000 100.000 99.604
323 100.000 100.000 100.000 100.000 99.606
324 100.000 100.000 100.000 100.000 99.610
325 100.000 100.000 100.000 100.000 99.614
326 100.000 100.000 100.000 100.000 99.615
327 100.000 100.000 100.000 100.000 99.617
328 100.000 100.000 100.000 100.000 99.619
329 100.000 100.000 100.000 100.000 99.621
330 100.000 100.000 100.000 100.000 99.623
331 100.000 100.000 100.000 100.000 99.625
332 100.000 100.000 100.000 100.000 99.626
333 100.000 100.000 100.000 100.000 99.636
334 100.000 100.000 100.000 100.000 99.638
335 100.000 100.000 100.000 100.000 99.639
336 100.000 100.000 100.000 100.000 99.641
337 100.000 100.000 100.000 100.000 99.645
338 100.000 100.000 100.000 100.000 99.647
339 100.000 100.000 100.000 100.000 99.650
340 100.000 100.000 100.000 100.000 99.654
341 100.000 100.000 100.000 100.000 99.658
342 100.000 100.000 100.000 100.000 99.665
343 100.000 100.000 100.000 100.000 99.673
344 100.000 100.000 100.000 100.000 99.675
345 100.000 100.000 100.000 100.000 99.678
346 100.000 100.000 100.000 100.000 99.680
347 100.000 100.000 100.000 100.000 99.684
348 100.000 100.000 100.000 100.000 99.686
349 100.000 100.000 100.000 100.000 99.687
350 100.000 100.000 100.000 100.000 99.689
351 100.000 100.000 100.000 100.000 99.691
352 100.000 100.000 100.000 100.000 99.695
353 100.000 100.000 100.000 100.000 99.697
354 100.000 100.000 100.000 100.000 99.700
355 100.000 100.000 100.000 100.000 99.704
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 59.524 32.487 18.442 11.077 7.033
4 64.286 37.563 22.548 14.101 9.181
5 71.429 43.993 27.241 17.345 11.402
6 76.190 49.577 31.699 20.535 13.614
7 85.714 56.853 36.603 23.806 15.826
8 88.095 60.406 39.934 26.487 17.843
9 90.476 64.129 43.383 29.221 19.879
10 92.857 66.836 46.011 31.494 21.702
11 92.857 69.205 48.780 33.902 23.590
12 92.857 71.743 51.783 36.486 25.575
13 92.857 74.112 54.528 38.867 27.444
14 95.238 76.819 57.062 41.033 29.192
15 95.238 78.680 59.362 43.134 30.913
16 95.238 79.526 60.770 44.689 32.358
17 95.238 80.372 62.224 46.292 33.828
18 97.619 83.418 64.969 48.512 35.547
19 97.619 84.772 66.729 50.212 37.018
20 100.000 86.971 68.724 51.965 38.497
21 100.000 87.986 70.202 53.498 39.884
22 100.000 89.002 71.633 54.962 41.204
23 100.000 89.679 72.736 56.206 42.404
24 100.000 90.525 74.003 57.557 43.660
25 100.000 91.032 74.988 58.748 44.842
26 100.000 91.878 76.255 60.099 46.097
27 100.000 92.217 77.030 61.117 47.168
28 100.000 92.724 77.921 62.174 48.228
29 100.000 93.232 78.789 63.198 49.259
30 100.000 93.570 79.493 64.115 50.236
31 100.000 94.078 80.338 65.112 51.247
32 100.000 94.247 80.878 65.911 52.148
33 100.000 94.247 81.300 66.656 53.030
34 100.000 94.247 81.628 67.289 53.831
35 100.000 94.755 82.567 68.393 54.909
36 100.000 94.924 83.153 69.251 55.848
37 100.000 95.431 83.951 70.168 56.775
38 100.000 95.601 84.444 70.897 57.605
39 100.000 95.939 85.054 71.664 58.428
40 100.000 96.277 85.687 72.457 59.265
41 100.000 96.447 86.086 73.052 59.977
42 100.000 96.616 86.532 73.701 60.725
43 100.000 96.616 86.861 74.296 61.449
44 100.000 97.293 87.823 75.304 62.393
45 100.000 97.293 88.104 75.830 63.051
46 100.000 97.293 88.339 76.296 63.663
47 100.000 97.462 88.738 76.870 64.330
48 100.000 97.800 89.301 77.551 65.052
49 100.000 97.970 89.700 78.119 65.708
50 100.000 98.646 90.591 79.020 66.546
51 100.000 98.816 90.967 79.556 67.175
52 100.000 98.816 91.178 79.980 67.739
53 100.000 98.816 91.389 80.403 68.303
54 100.000 98.816 91.553 80.757 68.800
55 100.000 98.816 91.741 81.138 69.314
56 100.000 98.816 91.882 81.454 69.775
57 100.000 98.816 92.093 81.872 70.320
58 100.000 98.985 92.398 82.301 70.840
59 100.000 99.154 92.726 82.757 71.378
60 100.000 99.154 92.891 83.100 71.851
61 100.000 99.154 93.055 83.443 72.325
62 100.000 99.323 93.407 83.926 72.879
63 100.000 99.323 93.595 84.296 73.369
64 100.000 99.323 93.759 84.634 73.832
65 100.000 99.492 94.111 85.111 74.374
66 100.000 99.492 94.181 85.320 74.725
67 100.000 99.492 94.275 85.556 75.093
68 100.000 99.492 94.416 85.851 75.503
69 100.000 99.492 94.533 86.113 75.888
70 100.000 99.492 94.627 86.349 76.252
71 100.000 99.492 94.744 86.612 76.635
72 100.000 99.662 95.073 87.052 77.131
73 100.000 99.662 95.167 87.277 77.477
74 100.000 99.662 95.260 87.502 77.821
75 100.000 99.662 95.401 87.786 78.207
76 100.000 99.662 95.542 88.065 78.582
77 100.000 99.662 95.683 88.349 78.961
78 100.000 99.662 95.777 88.564 79.281
79 100.000 99.662 95.870 88.773 79.592
80 100.000 99.662 95.988 89.014 79.929
81 100.000 99.662 96.082 89.223 80.237
82 100.000 99.662 96.199 89.464 80.572
83 100.000 99.831 96.504 89.850 80.996
84 100.000 99.831 96.598 90.060 81.304
85 100.000 99.831 96.668 90.236 81.586
86 100.000 99.831 96.739 90.419 81.874
87 100.000 99.831 96.832 90.622 82.172
88 100.000 99.831 96.926 90.826 82.468
89 100.000 99.831 96.997 90.992 82.726
90 100.000 99.831 97.067 91.159 82.984
91 100.000 99.831 97.161 91.357 83.266
92 100.000 99.831 97.231 91.529 83.529
93 100.000 99.831 97.278 91.657 83.749
94 100.000 99.831 97.349 91.823 84.002
95 100.000 99.831 97.396 91.952 84.221
96 100.000 99.831 97.443 92.081 84.439
97 100.000 99.831 97.536 92.274 84.707
98 100.000 99.831 97.630 92.467 84.975
99 100.000 99.831 97.677 92.596 85.190
100 100.000 99.831 97.677 92.660 85.352
101 100.000 99.831 97.724 92.783 85.558
102 100.000 99.831 97.771 92.907 85.763
103 100.000 99.831 97.841 93.062 85.994
104 100.000 99.831 97.888 93.185 86.199
105 100.000 99.831 97.959 93.341 86.429
106 100.000 99.831 98.029 93.496 86.658
107 100.000 99.831 98.099 93.652 86.887
108 100.000 99.831 98.146 93.775 87.089
109 100.000 99.831 98.146 93.839 87.244
110 100.000 99.831 98.170 93.925 87.411
111 100.000 100.000 98.428 94.220 87.716
112 100.000 100.000 98.475 94.338 87.906
113 100.000 100.000 98.522 94.456 88.097
114 100.000 100.000 98.592 94.606 88.313
115 100.000 100.000 98.663 94.756 88.529
116 100.000 100.000 98.710 94.874 88.720
117 100.000 100.000 98.780 95.024 88.936
118 100.000 100.000 98.780 95.078 89.075
119 100.000 100.000 98.850 95.228 89.291
120 100.000 100.000 98.850 95.282 89.430
121 100.000 100.000 98.874 95.362 89.583
122 100.000 100.000 98.921 95.475 89.763
123 100.000 100.000 98.968 95.587 89.940
124 100.000 100.000 99.015 95.700 90.118
125 100.000 100.000 99.061 95.813 90.295
126 100.000 100.000 99.085 95.888 90.438
127 100.000 100.000 99.085 95.931 90.552
128 100.000 100.000 99.132 96.038 90.719
129 100.000 100.000 99.179 96.145 90.883
130 100.000 100.000 99.226 96.252 91.048
131 100.000 100.000 99.273 96.359 91.213
132 100.000 100.000 99.273 96.402 91.323
133 100.000 100.000 99.296 96.472 91.453
134 100.000 100.000 99.320 96.542 91.581
135 100.000 100.000 99.343 96.611 91.708
136 100.000 100.000 99.366 96.681 91.834
137 100.000 100.000 99.413 96.783 91.986
138 100.000 100.000 99.437 96.853 92.111
139 100.000 100.000 99.460 96.922 92.237
140 100.000 100.000 99.484 96.992 92.363
141 100.000 100.000 99.531 97.094 92.513
142 100.000 100.000 99.554 97.158 92.629
143 100.000 100.000 99.554 97.190 92.720
144 100.000 100.000 99.601 97.292 92.869
145 100.000 100.000 99.648 97.394 93.019
146 100.000 100.000 99.648 97.421 93.102
147 100.000 100.000 99.648 97.453 93.189
148 100.000 100.000 99.672 97.518 93.302
149 100.000 100.000 99.672 97.550 93.389
150 100.000 100.000 99.672 97.577 93.465
151 100.000 100.000 99.672 97.603 93.541
152 100.000 100.000 99.672 97.630 93.615
153 100.000 100.000 99.695 97.689 93.715
154 100.000 100.000 99.718 97.748 93.814
155 100.000 100.000 99.718 97.775 93.888
156 100.000 100.000 99.742 97.834 93.986
157 100.000 100.000 99.742 97.861 94.058
158 100.000 100.000 99.765 97.920 94.156
159 100.000 100.000 99.789 97.979 94.254
160 100.000 100.000 99.812 98.038 94.352
161 100.000 100.000 99.836 98.097 94.451
162 100.000 100.000 99.859 98.156 94.549
163 100.000 100.000 99.859 98.182 94.621
164 100.000 100.000 99.859 98.209 94.693
165 100.000 100.000 99.883 98.268 94.791
166 100.000 100.000 99.883 98.295 94.863
167 100.000 100.000 99.883 98.322 94.935
168 100.000 100.000 99.883 98.343 94.998
169 100.000 100.000 99.883 98.365 95.061
170 100.000 100.000 99.883 98.386 95.124
171 100.000 100.000 99.883 98.408 95.186
172 100.000 100.000 99.883 98.429 95.249
173 100.000 100.000 99.906 98.483 95.336
174 100.000 100.000 99.906 98.504 95.397
175 100.000 100.000 99.906 98.526 95.458
176 100.000 100.000 99.906 98.536 95.505
177 100.000 100.000 99.906 98.558 95.564
178 100.000 100.000 99.906 98.579 95.623
179 100.000 100.000 99.906 98.601 95.682
180 100.000 100.000 99.906 98.622 95.741
181 100.000 100.000 99.906 98.644 95.800
182 100.000 100.000 99.906 98.665 95.860
183 100.000 100.000 99.930 98.719 95.945
184 100.000 100.000 99.930 98.740 96.004
185 100.000 100.000 99.953 98.794 96.089
186 100.000 100.000 99.953 98.810 96.141
187 100.000 100.000 99.953 98.831 96.200
188 100.000 100.000 99.953 98.853 96.259
189 100.000 100.000 99.953 98.874 96.318
190 100.000 100.000 99.953 98.896 96.377
191 100.000 100.000 99.953 98.917 96.437
192 100.000 100.000 99.977 98.971 96.520
193 100.000 100.000 100.000 99.024 96.603
194 100.000 100.000 100.000 99.040 96.653
195 100.000 100.000 100.000 99.056 96.701
196 100.000 100.000 100.000 99.072 96.749
197 100.000 100.000 100.000 99.089 96.797
198 100.000 100.000 100.000 99.105 96.845
199 100.000 100.000 100.000 99.121 96.893
200 100.000 100.000 100.000 99.137 96.940
201 100.000 100.000 100.000 99.153 96.986
202 100.000 100.000 100.000 99.169 97.032
203 100.000 100.000 100.000 99.185 97.078
204 100.000 100.000 100.000 99.201 97.124
205 100.000 100.000 100.000 99.217 97.171
206 100.000 100.000 100.000 99.233 97.217
207 100.000 100.000 100.000 99.249 97.263
208 100.000 100.000 100.000 99.265 97.309
209 100.000 100.000 100.000 99.282 97.354
210 100.000 100.000 100.000 99.292 97.389
211 100.000 100.000 100.000 99.303 97.424
212 100.000 100.000 100.000 99.314 97.459
213 100.000 100.000 100.000 99.324 97.492
214 100.000 100.000 100.000 99.335 97.526
215 100.000 100.000 100.000 99.346 97.559
216 100.000 100.000 100.000 99.357 97.592
217 100.000 100.000 100.000 99.367 97.626
218 100.000 100.000 100.000 99.378 97.659
219 100.000 100.000 100.000 99.389 97.690
220 100.000 100.000 100.000 99.399 97.722
221 100.000 100.000 100.000 99.410 97.753
222 100.000 100.000 100.000 99.421 97.785
223 100.000 100.000 100.000 99.432 97.816
224 100.000 100.000 100.000 99.442 97.848
225 100.000 100.000 100.000 99.453 97.879
226 100.000 100.000 100.000 99.464 97.910
227 100.000 100.000 100.000 99.475 97.942
228 100.000 100.000 100.000 99.485 97.973
229 100.000 100.000 100.000 99.496 98.005
230 100.000 100.000 100.000 99.507 98.036
231 100.000 100.000 100.000 99.517 98.068
232 100.000 100.000 100.000 99.528 98.099
233 100.000 100.000 100.000 99.539 98.130
234 100.000 100.000 100.000 99.550 98.162
235 100.000 100.000 100.000 99.560 98.193
236 100.000 100.000 100.000 99.571 98.225
237 100.000 100.000 100.000 99.582 98.256
238 100.000 100.000 100.000 99.593 98.288
239 100.000 100.000 100.000 99.603 98.319
240 100.000 100.000 100.000 99.614 98.351
241 100.000 100.000 100.000 99.625 98.382
242 100.000 100.000 100.000 99.635 98.413
243 100.000 100.000 100.000 99.646 98.445
244 100.000 100.000 100.000 99.657 98.476
245 100.000 100.000 100.000 99.662 98.497
246 100.000 100.000 100.000 99.668 98.517
247 100.000 100.000 100.000 99.673 98.535
248 100.000 100.000 100.000 99.678 98.554
249 100.000 100.000 100.000 99.684 98.572
250 100.000 100.000 100.000 99.689 98.591
251 100.000 100.000 100.000 99.694 98.609
252 100.000 100.000 100.000 99.700 98.628
253 100.000 100.000 100.000 99.705 98.645
254 100.000 100.000 100.000 99.710 98.661
255 100.000 100.000 100.000 99.710 98.670
256 100.000 100.000 100.000 99.716 98.687
257 100.000 100.000 100.000 99.721 98.704
258 100.000 100.000 100.000 99.727 98.720
259 100.000 100.000 100.000 99.732 98.737
260 100.000 100.000 100.000 99.737 98.754
261 100.000 100.000 100.000 99.743 98.770
262 100.000 100.000 100.000 99.748 98.787
263 100.000 100.000 100.000 99.753 98.804
264 100.000 100.000 100.000 99.759 98.820
265 100.000 100.000 100.000 99.764 98.837
266 100.000 100.000 100.000 99.769 98.853
267 100.000 100.000 100.000 99.775 98.870
268 100.000 100.000 100.000 99.780 98.887
269 100.000 100.000 100.000 99.786 98.903
270 100.000 100.000 100.000 99.791 98.920
271 100.000 100.000 100.000 99.796 98.937
272 100.000 100.000 100.000 99.802 98.953
273 100.000 100.000 100.000 99.807 98.970
274 100.000 100.000 100.000 99.812 98.987
275 100.000 100.000 100.000 99.818 99.003
276 100.000 100.000 100.000 99.823 99.020
277 100.000 100.000 100.000 99.828 99.037
278 100.000 100.000 100.000 99.834 99.053
279 100.000 100.000 100.000 99.839 99.070
280 100.000 100.000 100.000 99.845 99.086
281 100.000 100.000 100.000 99.850 99.103
282 100.000 100.000 100.000 99.855 99.120
283 100.000 100.000 100.000 99.861 99.136
284 100.000 100.000 100.000 99.866 99.153
285 100.000 100.000 100.000 99.871 99.170
286 100.000 100.000 100.000 99.877 99.186
287 100.000 100.000 100.000 99.882 99.203
288 100.000 100.000 100.000 99.887 99.220
289 100.000 100.000 100.000 99.893 99.236
290 100.000 100.000 100.000 99.898 99.253
291 100.000 100.000 100.000 99.903 99.270
292 100.000 100.000 100.000 99.909 99.286
293 100.000 100.000 100.000 99.914 99.303
294 100.000 100.000 100.000 99.920 99.319
295 100.000 100.000 100.000 99.925 99.336
296 100.000 100.000 100.000 99.930 99.353
297 100.000 100.000 100.000 99.936 99.369
298 100.000 100.000 100.000 99.941 99.386
299 100.000 100.000 100.000 99.946 99.403
300 100.000 100.000 100.000 99.952 99.419
301 100.000 100.000 100.000 99.957 99.436
302 100.000 100.000 100.000 99.962 99.453
303 100.000 100.000 100.000 99.968 99.469
304 100.000 100.000 100.000 99.973 99.486
305 100.000 100.000 100.000 99.979 99.503
306 100.000 100.000 100.000 99.984 99.519
307 100.000 100.000 100.000 99.989 99.536
308 100.000 100.000 100.000 99.995 99.552
309 100.000 100.000 100.000 100.000 99.569
310 100.000 100.000 100.000 100.000 99.577
311 100.000 100.000 100.000 100.000 99.584
312 100.000 100.000 100.000 100.000 99.591
313 100.000 100.000 100.000 100.000 99.597
314 100.000 100.000 100.000 100.000 99.601
315 100.000 100.000 100.000 100.000 99.604
316 100.000 100.000 100.000 100.000 99.608
317 100.000 100.000 100.000 100.000 99.612
318 100.000 100.000 100.000 100.000 99.615
319 100.000 100.000 100.000 100.000 99.619
320 100.000 100.000 100.000 100.000 99.623
321 100.000 100.000 100.000 100.000 99.626
322 100.000 100.000 100.000 100.000 99.630
323 100.000 100.000 100.000 100.000 99.634
324 100.000 100.000 100.000 100.000 99.638
325 100.000 100.000 100.000 100.000 99.641
326 100.000 100.000 100.000 100.000 99.645
327 100.000 100.000 100.000 100.000 99.649
328 100.000 100.000 100.000 100.000 99.652
329 100.000 100.000 100.000 100.000 99.656
330 100.000 100.000 100.000 100.000 99.658
331 100.000 100.000 100.000 100.000 99.660
332 100.000 100.000 100.000 100.000 99.662
333 100.000 100.000 100.000 100.000 99.663
334 100.000 100.000 100.000 100.000 99.665
335 100.000 100.000 100.000 100.000 99.667
336 100.000 100.000 100.000 100.000 99.669
337 100.000 100.000 100.000 100.000 99.671
338 100.000 100.000 100.000 100.000 99.673
339 100.000 100.000 100.000 100.000 99.675
340 100.000 100.000 100.000 100.000 99.676
341 100.000 100.000 100.000 100.000 99.678
342 100.000 100.000 100.000 100.000 99.680
343 100.000 100.000 100.000 100.000 99.682
344 100.000 100.000 100.000 100.000 99.684
345 100.000 100.000 100.000 100.000 99.686
346 100.000 100.000 100.000 100.000 99.687
347 100.000 100.000 100.000 100.000 99.689
348 100.000 100.000 100.000 100.000 99.691
349 100.000 100.000 100.000 100.000 99.693
350 100.000 100.000 100.000 100.000 99.695
351 100.000 100.000 100.000 100.000 99.697
352 100.000 100.000 100.000 100.000 99.699
353 100.000 100.000 100.000 100.000 99.700
354 100.000 100.000 100.000 100.000 99.702
355 100.000 100.000 100.000 100.000 99.704
356 100.000 100.000 100.000 100.000 99.706

grep

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 50.000 24.704 13.186 7.640 4.758
3 61.905 33.841 19.052 11.318 7.116
4 73.810 42.809 24.801 14.943 9.455
5 83.333 50.761 30.221 18.482 11.778
6 85.714 55.499 34.444 21.618 13.980
7 92.857 61.421 38.691 24.588 16.062
8 95.238 65.990 42.820 27.693 18.257
9 97.619 70.051 46.574 30.610 20.378
10 100.000 73.773 49.883 33.156 22.255
11 100.000 75.804 52.346 35.365 24.032
12 100.000 77.834 54.974 37.773 25.959
13 100.000 79.865 57.461 40.025 27.781
14 100.000 81.557 59.596 42.008 29.427
15 100.000 82.910 61.427 43.805 30.982
16 100.000 84.264 63.257 45.590 32.517
17 100.000 85.618 65.275 47.601 34.223
18 100.000 86.802 66.964 49.327 35.758
19 100.000 87.817 68.419 50.818 37.097
20 100.000 88.832 69.826 52.228 38.345
21 100.000 89.679 71.187 53.708 39.710
22 100.000 90.525 72.478 55.096 41.004
23 100.000 91.371 73.698 56.367 42.169
24 100.000 92.217 74.918 57.648 43.359
25 100.000 92.893 75.903 58.699 44.353
26 100.000 93.570 76.983 59.906 45.515
27 100.000 94.078 77.992 61.123 46.715
28 100.000 94.585 78.930 62.243 47.830
29 100.000 95.093 79.775 63.214 48.788
30 100.000 95.601 80.690 64.308 49.888
31 100.000 96.108 81.417 65.074 50.609
32 100.000 96.447 82.309 66.243 51.800
33 100.000 96.785 82.802 66.779 52.327
34 100.000 97.124 83.623 67.836 53.400
35 100.000 97.462 84.280 68.656 54.254
36 100.000 97.800 84.984 69.557 55.199
37 100.000 98.139 85.476 70.103 55.750
38 100.000 98.477 85.969 70.640 56.277
39 100.000 98.646 86.368 71.192 56.897
40 100.000 98.816 86.649 71.540 57.281
41 100.000 98.985 87.236 72.398 58.221
42 100.000 99.154 87.565 72.838 58.716
43 100.000 99.323 88.104 73.626 59.593
44 100.000 99.492 88.456 74.114 60.159
45 100.000 99.662 88.761 74.500 60.582
46 100.000 99.831 89.019 74.795 60.887
47 100.000 100.000 89.277 75.090 61.192
48 100.000 100.000 89.465 75.433 61.618
49 100.000 100.000 89.676 75.830 62.123
50 100.000 100.000 89.887 76.248 62.672
51 100.000 100.000 90.099 76.661 63.212
52 100.000 100.000 90.216 76.897 63.532
53 100.000 100.000 90.333 77.133 63.852
54 100.000 100.000 90.544 77.556 64.416
55 100.000 100.000 90.591 77.679 64.617
56 100.000 100.000 90.779 78.044 65.094
57 100.000 100.000 90.967 78.409 65.571
58 100.000 100.000 91.154 78.763 66.021
59 100.000 100.000 91.248 78.966 66.313
60 100.000 100.000 91.412 79.342 66.851
61 100.000 100.000 91.483 79.502 67.091
62 100.000 100.000 91.835 80.157 67.899
63 100.000 100.000 92.163 80.752 68.624
64 100.000 100.000 92.210 80.870 68.815
65 100.000 100.000 92.374 81.207 69.277
66 100.000 100.000 92.445 81.363 69.506
67 100.000 100.000 92.656 81.781 70.058
68 100.000 100.000 92.750 81.974 70.327
69 100.000 100.000 92.961 82.382 70.856
70 100.000 100.000 93.008 82.484 71.006
71 100.000 100.000 93.219 82.929 71.611
72 100.000 100.000 93.360 83.202 71.970
73 100.000 100.000 93.430 83.368 72.227
74 100.000 100.000 93.430 83.433 72.382
75 100.000 100.000 93.454 83.497 72.493
76 100.000 100.000 93.524 83.642 72.694
77 100.000 100.000 93.618 83.845 72.990
78 100.000 100.000 93.665 83.979 73.216
79 100.000 100.000 93.853 84.333 73.665
80 100.000 100.000 94.134 84.848 74.300
81 100.000 100.000 94.228 85.062 74.620
82 100.000 100.000 94.298 85.234 74.886
83 100.000 100.000 94.345 85.347 75.063
84 100.000 100.000 94.369 85.406 75.161
85 100.000 100.000 94.416 85.518 75.339
86 100.000 100.000 94.510 85.717 75.618
87 100.000 100.000 94.603 85.899 75.858
88 100.000 100.000 94.603 85.942 75.969
89 100.000 100.000 94.603 85.958 76.016
90 100.000 100.000 94.627 86.033 76.154
91 100.000 100.000 94.791 86.371 76.617
92 100.000 100.000 94.791 86.398 76.689
93 100.000 100.000 94.838 86.521 76.894
94 100.000 100.000 94.908 86.676 77.123
95 100.000 100.000 94.955 86.789 77.301
96 100.000 100.000 95.049 86.982 77.569
97 100.000 100.000 95.096 87.116 77.795
98 100.000 100.000 95.143 87.223 77.957
99 100.000 100.000 95.167 87.282 78.055
100 100.000 100.000 95.260 87.475 78.324
101 100.000 100.000 95.448 87.845 78.814
102 100.000 100.000 95.495 87.974 79.026
103 100.000 100.000 95.612 88.226 79.385
104 100.000 100.000 95.659 88.349 79.587
105 100.000 100.000 95.753 88.564 79.899
106 100.000 100.000 95.753 88.596 79.990
107 100.000 100.000 95.777 88.665 80.115
108 100.000 100.000 95.824 88.778 80.293
109 100.000 100.000 95.824 88.853 80.472
110 100.000 100.000 95.847 88.917 80.583
111 100.000 100.000 95.894 89.030 80.761
112 100.000 100.000 95.964 89.202 81.025
113 100.000 100.000 96.082 89.443 81.360
114 100.000 100.000 96.082 89.486 81.471
115 100.000 100.000 96.129 89.598 81.648
116 100.000 100.000 96.129 89.614 81.693
117 100.000 100.000 96.152 89.673 81.791
118 100.000 100.000 96.152 89.684 81.822
119 100.000 100.000 96.222 89.829 82.024
120 100.000 100.000 96.246 89.888 82.125
121 100.000 100.000 96.340 90.092 82.418
122 100.000 100.000 96.410 90.242 82.634
123 100.000 100.000 96.410 90.269 82.708
124 100.000 100.000 96.457 90.370 82.858
125 100.000 100.000 96.527 90.521 83.076
126 100.000 100.000 96.574 90.622 83.229
127 100.000 100.000 96.598 90.676 83.313
128 100.000 100.000 96.598 90.730 83.448
129 100.000 100.000 96.668 90.891 83.692
130 100.000 100.000 96.715 90.998 83.855
131 100.000 100.000 96.786 91.143 84.058
132 100.000 100.000 96.809 91.202 84.158
133 100.000 100.000 96.832 91.261 84.256
134 100.000 100.000 96.832 91.271 84.287
135 100.000 100.000 96.856 91.325 84.370
136 100.000 100.000 96.856 91.346 84.430
137 100.000 100.000 96.856 91.357 84.461
138 100.000 100.000 96.879 91.421 84.572
139 100.000 100.000 96.903 91.502 84.722
140 100.000 100.000 96.950 91.614 84.905
141 100.000 100.000 96.950 91.625 84.936
142 100.000 100.000 96.997 91.727 85.086
143 100.000 100.000 97.020 91.791 85.197
144 100.000 100.000 97.044 91.845 85.280
145 100.000 100.000 97.067 91.899 85.363
146 100.000 100.000 97.114 92.011 85.541
147 100.000 100.000 97.231 92.242 85.848
148 100.000 100.000 97.278 92.365 86.050
149 100.000 100.000 97.278 92.381 86.094
150 100.000 100.000 97.325 92.499 86.286
151 100.000 100.000 97.372 92.606 86.449
152 100.000 100.000 97.443 92.767 86.689
153 100.000 100.000 97.443 92.815 86.813
154 100.000 100.000 97.466 92.869 86.898
155 100.000 100.000 97.466 92.885 86.945
156 100.000 100.000 97.466 92.896 86.976
157 100.000 100.000 97.489 92.949 87.059
158 100.000 100.000 97.489 92.960 87.091
159 100.000 100.000 97.513 93.041 87.242
160 100.000 100.000 97.536 93.100 87.340
161 100.000 100.000 97.536 93.121 87.399
162 100.000 100.000 97.536 93.142 87.459
163 100.000 100.000 97.607 93.293 87.675
164 100.000 100.000 97.607 93.298 87.692
165 100.000 100.000 97.630 93.352 87.775
166 100.000 100.000 97.724 93.550 88.056
167 100.000 100.000 97.724 93.604 88.191
168 100.000 100.000 97.724 93.614 88.222
169 100.000 100.000 97.748 93.689 88.361
170 100.000 100.000 97.771 93.743 88.444
171 100.000 100.000 97.818 93.845 88.594
172 100.000 100.000 97.818 93.877 88.681
173 100.000 100.000 97.818 93.898 88.740
174 100.000 100.000 97.841 93.952 88.823
175 100.000 100.000 97.888 94.059 88.988
176 100.000 100.000 97.888 94.070 89.019
177 100.000 100.000 97.888 94.070 89.023
178 100.000 100.000 97.888 94.081 89.054
179 100.000 100.000 97.888 94.102 89.117
180 100.000 100.000 97.888 94.129 89.189
181 100.000 100.000 97.912 94.188 89.289
182 100.000 100.000 97.912 94.188 89.291
183 100.000 100.000 97.912 94.204 89.339
184 100.000 100.000 97.912 94.226 89.400
185 100.000 100.000 97.912 94.236 89.432
186 100.000 100.000 97.912 94.236 89.434
187 100.000 100.000 97.912 94.242 89.450
188 100.000 100.000 97.912 94.258 89.496
189 100.000 100.000 97.912 94.258 89.504
190 100.000 100.000 97.935 94.311 89.591
191 100.000 100.000 97.935 94.322 89.622
192 100.000 100.000 97.935 94.333 89.654
193 100.000 100.000 98.006 94.483 89.870
194 100.000 100.000 98.099 94.676 90.138
195 100.000 100.000 98.099 94.687 90.170
196 100.000 100.000 98.099 94.719 90.260
197 100.000 100.000 98.123 94.778 90.358
198 100.000 100.000 98.170 94.880 90.508
199 100.000 100.000 98.193 94.933 90.591
200 100.000 100.000 98.264 95.078 90.793
201 100.000 100.000 98.358 95.271 91.061
202 100.000 100.000 98.358 95.287 91.109
203 100.000 100.000 98.381 95.351 91.220
204 100.000 100.000 98.428 95.480 91.434
205 100.000 100.000 98.451 95.534 91.518
206 100.000 100.000 98.451 95.555 91.577
207 100.000 100.000 98.451 95.566 91.612
208 100.000 100.000 98.451 95.587 91.671
209 100.000 100.000 98.451 95.587 91.679
210 100.000 100.000 98.451 95.587 91.680
211 100.000 100.000 98.451 95.603 91.727
212 100.000 100.000 98.522 95.754 91.943
213 100.000 100.000 98.522 95.759 91.961
214 100.000 100.000 98.592 95.904 92.163
215 100.000 100.000 98.592 95.925 92.222
216 100.000 100.000 98.592 95.936 92.254
217 100.000 100.000 98.592 95.941 92.272
218 100.000 100.000 98.592 95.947 92.289
219 100.000 100.000 98.592 95.952 92.307
220 100.000 100.000 98.592 95.952 92.315
221 100.000 100.000 98.592 95.963 92.350
222 100.000 100.000 98.616 96.016 92.433
223 100.000 100.000 98.616 96.022 92.450
224 100.000 100.000 98.663 96.124 92.599
225 100.000 100.000 98.663 96.140 92.646
226 100.000 100.000 98.663 96.156 92.692
227 100.000 100.000 98.710 96.258 92.842
228 100.000 100.000 98.710 96.258 92.849
229 100.000 100.000 98.710 96.263 92.866
230 100.000 100.000 98.710 96.274 92.897
231 100.000 100.000 98.710 96.295 92.956
232 100.000 100.000 98.733 96.359 93.069
233 100.000 100.000 98.733 96.365 93.086
234 100.000 100.000 98.733 96.376 93.117
235 100.000 100.000 98.733 96.381 93.134
236 100.000 100.000 98.756 96.435 93.219
237 100.000 100.000 98.756 96.445 93.250
238 100.000 100.000 98.756 96.461 93.295
239 100.000 100.000 98.780 96.520 93.393
240 100.000 100.000 98.780 96.526 93.411
241 100.000 100.000 98.803 96.585 93.509
242 100.000 100.000 98.803 96.611 93.581
243 100.000 100.000 98.827 96.670 93.679
244 100.000 100.000 98.827 96.670 93.683
245 100.000 100.000 98.827 96.692 93.742
246 100.000 100.000 98.827 96.703 93.774
247 100.000 100.000 98.827 96.713 93.805
248 100.000 100.000 98.827 96.724 93.837
249 100.000 100.000 98.827 96.729 93.853
250 100.000 100.000 98.827 96.745 93.898
251 100.000 100.000 98.850 96.799 93.981
252 100.000 100.000 98.850 96.831 94.071
253 100.000 100.000 98.850 96.837 94.092
254 100.000 100.000 98.874 96.890 94.175
255 100.000 100.000 98.874 96.912 94.236
256 100.000 100.000 98.874 96.922 94.267
257 100.000 100.000 98.874 96.928 94.284
258 100.000 100.000 98.874 96.939 94.327
259 100.000 100.000 98.874 96.949 94.358
260 100.000 100.000 98.874 96.960 94.389
261 100.000 100.000 98.874 96.965 94.408
262 100.000 100.000 98.874 96.971 94.425
263 100.000 100.000 98.874 96.976 94.441
264 100.000 100.000 98.874 96.987 94.473
265 100.000 100.000 98.874 96.997 94.504
266 100.000 100.000 98.897 97.062 94.615
267 100.000 100.000 98.897 97.083 94.674
268 100.000 100.000 98.897 97.083 94.678
269 100.000 100.000 98.897 97.094 94.709
270 100.000 100.000 98.897 97.094 94.711
271 100.000 100.000 98.897 97.099 94.728
272 100.000 100.000 98.897 97.105 94.745
273 100.000 100.000 98.944 97.207 94.894
274 100.000 100.000 98.944 97.212 94.911
275 100.000 100.000 98.944 97.217 94.929
276 100.000 100.000 98.944 97.217 94.933
277 100.000 100.000 98.968 97.271 95.016
278 100.000 100.000 98.968 97.271 95.018
279 100.000 100.000 98.968 97.271 95.026
280 100.000 100.000 98.991 97.325 95.109
281 100.000 100.000 98.991 97.335 95.140
282 100.000 100.000 98.991 97.351 95.185
283 100.000 100.000 98.991 97.351 95.192
284 100.000 100.000 98.991 97.351 95.198
285 100.000 100.000 98.991 97.357 95.214
286 100.000 100.000 98.991 97.357 95.222
287 100.000 100.000 98.991 97.362 95.238
288 100.000 100.000 98.991 97.362 95.240
289 100.000 100.000 99.015 97.416 95.323
290 100.000 100.000 99.015 97.426 95.355
291 100.000 100.000 99.015 97.432 95.371
292 100.000 100.000 99.015 97.442 95.405
293 100.000 100.000 99.015 97.448 95.421
294 100.000 100.000 99.038 97.507 95.519
295 100.000 100.000 99.273 97.957 96.091
296 100.000 100.000 99.273 97.968 96.122
297 100.000 100.000 99.296 98.022 96.205
298 100.000 100.000 99.296 98.032 96.239
299 100.000 100.000 99.296 98.038 96.259
300 100.000 100.000 99.296 98.048 96.290
301 100.000 100.000 99.296 98.054 96.307
302 100.000 100.000 99.296 98.054 96.311
303 100.000 100.000 99.296 98.054 96.313
304 100.000 100.000 99.296 98.064 96.344
305 100.000 100.000 99.296 98.081 96.388
306 100.000 100.000 99.296 98.091 96.420
307 100.000 100.000 99.296 98.091 96.422
308 100.000 100.000 99.296 98.097 96.438
309 100.000 100.000 99.320 98.150 96.522
310 100.000 100.000 99.320 98.156 96.538
311 100.000 100.000 99.320 98.161 96.555
312 100.000 100.000 99.320 98.161 96.557
313 100.000 100.000 99.320 98.166 96.573
314 100.000 100.000 99.320 98.166 96.577
315 100.000 100.000 99.320 98.177 96.609
316 100.000 100.000 99.320 98.188 96.644
317 100.000 100.000 99.320 98.188 96.646
318 100.000 100.000 99.320 98.188 96.653
319 100.000 100.000 99.320 98.193 96.671
320 100.000 100.000 99.320 98.193 96.673
321 100.000 100.000 99.320 98.209 96.718
322 100.000 100.000 99.531 98.627 97.263
323 100.000 100.000 99.531 98.627 97.271
324 100.000 100.000 99.554 98.681 97.354
325 100.000 100.000 99.554 98.692 97.385
326 100.000 100.000 99.578 98.745 97.470
327 100.000 100.000 99.578 98.745 97.476
328 100.000 100.000 99.578 98.756 97.507
329 100.000 100.000 99.578 98.756 97.511
330 100.000 100.000 99.578 98.767 97.542
331 100.000 100.000 99.601 98.820 97.626
332 100.000 100.000 99.601 98.820 97.627
333 100.000 100.000 99.601 98.826 97.644
334 100.000 100.000 99.601 98.826 97.646
335 100.000 100.000 99.601 98.842 97.692
336 100.000 100.000 99.601 98.847 97.709
337 100.000 100.000 99.601 98.847 97.714
338 100.000 100.000 99.601 98.858 97.750
339 100.000 100.000 99.601 98.863 97.766
340 100.000 100.000 99.601 98.863 97.774
341 100.000 100.000 99.601 98.869 97.790
342 100.000 100.000 99.601 98.869 97.794
343 100.000 100.000 99.601 98.874 97.811
344 100.000 100.000 99.601 98.879 97.827
345 100.000 100.000 99.601 98.879 97.831
346 100.000 100.000 99.601 98.890 97.862
347 100.000 100.000 99.601 98.901 97.894
348 100.000 100.000 99.601 98.906 97.910
349 100.000 100.000 99.601 98.912 97.927
350 100.000 100.000 99.601 98.917 97.944
351 100.000 100.000 99.601 98.917 97.947
352 100.000 100.000 99.601 98.928 97.979
353 100.000 100.000 99.601 98.933 97.995
354 100.000 100.000 99.601 98.944 98.027
355 100.000 100.000 99.601 98.949 98.044
356 100.000 100.000 99.601 98.954 98.060
357 100.000 100.000 99.648 99.056 98.210
358 100.000 100.000 99.789 99.341 98.589
359 100.000 100.000 99.789 99.346 98.606
360 100.000 100.000 99.789 99.346 98.609
361 100.000 100.000 99.789 99.351 98.626
362 100.000 100.000 99.789 99.351 98.630
363 100.000 100.000 99.789 99.351 98.633
364 100.000 100.000 99.789 99.357 98.654
365 100.000 100.000 99.789 99.362 98.672
366 100.000 100.000 99.789 99.362 98.674
367 100.000 100.000 99.789 99.362 98.676
368 100.000 100.000 99.789 99.362 98.680
369 100.000 100.000 99.789 99.373 98.715
370 100.000 100.000 99.789 99.378 98.731
371 100.000 100.000 99.789 99.383 98.748
372 100.000 100.000 99.789 99.389 98.765
373 100.000 100.000 99.789 99.389 98.768
374 100.000 100.000 99.789 99.394 98.785
375 100.000 100.000 99.812 99.448 98.868
376 100.000 100.000 99.812 99.448 98.872
377 100.000 100.000 99.812 99.448 98.874
378 100.000 100.000 99.812 99.485 98.970
379 100.000 100.000 99.812 99.485 98.974
380 100.000 100.000 99.812 99.485 98.976
381 100.000 100.000 99.812 99.496 99.007
382 100.000 100.000 99.812 99.496 99.011
383 100.000 100.000 99.812 99.496 99.013
384 100.000 100.000 99.812 99.501 99.029
385 100.000 100.000 99.812 99.501 99.033
386 100.000 100.000 99.812 99.501 99.037
387 100.000 100.000 99.812 99.501 99.040
388 100.000 100.000 99.812 99.501 99.044
389 100.000 100.000 99.812 99.507 99.061
390 100.000 100.000 99.812 99.512 99.077
391 100.000 100.000 99.883 99.662 99.294
392 100.000 100.000 99.883 99.668 99.310
393 100.000 100.000 99.883 99.668 99.314
394 100.000 100.000 99.930 99.769 99.464
395 100.000 100.000 99.930 99.769 99.466
396 100.000 100.000 99.930 99.769 99.467
397 100.000 100.000 99.930 99.780 99.499
398 100.000 100.000 99.930 99.780 99.501
399 100.000 100.000 99.930 99.780 99.504
400 100.000 100.000 99.930 99.780 99.508
401 100.000 100.000 99.930 99.780 99.510
402 100.000 100.000 99.930 99.786 99.527
403 100.000 100.000 99.930 99.791 99.543
404 100.000 100.000 99.930 99.791 99.545
405 100.000 100.000 99.930 99.796 99.562
406 100.000 100.000 99.930 99.796 99.565
407 100.000 100.000 99.930 99.796 99.567
408 100.000 100.000 99.930 99.796 99.571
409 100.000 100.000 99.930 99.802 99.588
410 100.000 100.000 99.930 99.802 99.589
411 100.000 100.000 99.930 99.802 99.593
412 100.000 100.000 99.930 99.802 99.597
413 100.000 100.000 99.930 99.802 99.601
414 100.000 100.000 99.930 99.812 99.632
415 100.000 100.000 99.930 99.818 99.649
416 100.000 100.000 99.930 99.818 99.650
417 100.000 100.000 99.930 99.818 99.652
418 100.000 100.000 99.930 99.823 99.669
419 100.000 100.000 99.930 99.834 99.700
420 100.000 100.000 99.930 99.839 99.717
421 100.000 100.000 99.930 99.845 99.734
422 100.000 100.000 99.930 99.845 99.736
423 100.000 100.000 99.930 99.845 99.739
424 100.000 100.000 99.930 99.845 99.741
425 100.000 100.000 99.930 99.845 99.745
426 100.000 100.000 99.930 99.845 99.747
427 100.000 100.000 99.930 99.845 99.750
428 100.000 100.000 99.930 99.845 99.752
429 100.000 100.000 99.930 99.845 99.756
430 100.000 100.000 99.930 99.845 99.758
431 100.000 100.000 99.930 99.845 99.760
432 100.000 100.000 99.930 99.845 99.761
433 100.000 100.000 99.930 99.845 99.765
434 100.000 100.000 99.930 99.845 99.767
435 100.000 100.000 100.000 99.995 99.983
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 61.905 33.841 19.052 11.318 7.116
4 71.429 41.794 24.472 14.857 9.438
5 78.571 48.393 29.329 18.203 11.707
6 80.952 52.623 33.247 21.221 13.877
7 85.714 58.037 37.588 24.358 16.070
8 90.476 63.283 41.741 27.355 18.180
9 95.238 68.020 45.519 30.165 20.216
10 95.238 70.728 48.592 32.760 22.196
11 97.619 74.112 51.760 35.328 24.149
12 100.000 76.481 54.223 37.564 25.976
13 100.000 78.511 56.710 39.805 27.781
14 100.000 79.865 58.775 41.874 29.534
15 100.000 81.557 60.934 43.906 31.235
16 100.000 83.249 63.069 45.906 32.909
17 100.000 84.772 65.063 47.820 34.538
18 100.000 86.125 66.823 49.509 35.988
19 100.000 87.140 68.325 51.096 37.439
20 100.000 88.156 69.803 52.608 38.785
21 100.000 88.832 71.023 54.013 40.126
22 100.000 89.679 72.313 55.418 41.450
23 100.000 90.694 73.698 56.833 42.741
24 100.000 91.540 74.965 58.185 43.997
25 100.000 92.047 75.997 59.455 45.265
26 100.000 92.724 77.053 60.629 46.399
27 100.000 93.570 78.203 61.793 47.458
28 100.000 94.078 79.141 62.924 48.594
29 100.000 94.585 79.986 63.895 49.552
30 100.000 94.755 80.643 64.854 50.593
31 100.000 94.924 81.276 65.777 51.586
32 100.000 95.093 81.910 66.726 52.636
33 100.000 95.601 82.708 67.626 53.527
34 100.000 96.108 83.505 68.533 54.430
35 100.000 96.616 84.280 69.406 55.303
36 100.000 96.954 84.960 70.270 56.207
37 100.000 97.462 85.687 71.085 57.026
38 100.000 97.970 86.415 71.851 57.747
39 100.000 98.308 87.048 72.640 58.574
40 100.000 98.308 87.471 73.374 59.432
41 100.000 98.646 88.081 74.157 60.286
42 100.000 98.816 88.550 74.865 61.113
43 100.000 98.985 89.019 75.540 61.875
44 100.000 99.154 89.489 76.221 62.648
45 100.000 99.154 89.840 76.848 63.400
46 100.000 99.154 90.169 77.444 64.133
47 100.000 99.323 90.568 78.001 64.765
48 100.000 99.323 90.873 78.564 65.462
49 100.000 99.323 91.178 79.127 66.159
50 100.000 99.662 91.671 79.663 66.686
51 100.000 99.662 91.952 80.199 67.358
52 100.000 99.662 92.187 80.666 67.972
53 100.000 99.831 92.515 81.106 68.467
54 100.000 99.831 92.726 81.524 69.020
55 100.000 99.831 92.938 81.947 69.584
56 100.000 99.831 93.149 82.366 70.130
57 100.000 100.000 93.454 82.752 70.553
58 100.000 100.000 93.641 83.138 71.078
59 100.000 100.000 93.829 83.513 71.579
60 100.000 100.000 93.993 83.840 72.018
61 100.000 100.000 94.158 84.178 72.480
62 100.000 100.000 94.322 84.521 72.955
63 100.000 100.000 94.486 84.843 73.381
64 100.000 100.000 94.650 85.170 73.819
65 100.000 100.000 94.815 85.502 74.268
66 100.000 100.000 94.955 85.786 74.647
67 100.000 100.000 95.096 86.092 75.082
68 100.000 100.000 95.237 86.381 75.481
69 100.000 100.000 95.378 86.666 75.864
70 100.000 100.000 95.519 86.971 76.299
71 100.000 100.000 95.636 87.218 76.643
72 100.000 100.000 95.753 87.481 77.027
73 100.000 100.000 95.870 87.738 77.399
74 100.000 100.000 95.988 87.990 77.759
75 100.000 100.000 96.105 88.247 78.131
76 100.000 100.000 96.222 88.489 78.462
77 100.000 100.000 96.340 88.730 78.793
78 100.000 100.000 96.457 88.982 79.154
79 100.000 100.000 96.551 89.191 79.459
80 100.000 100.000 96.645 89.373 79.699
81 100.000 100.000 96.739 89.620 80.091
82 100.000 100.000 96.832 89.824 80.389
83 100.000 100.000 96.926 90.017 80.657
84 100.000 100.000 97.020 90.215 80.936
85 100.000 100.000 97.114 90.440 81.280
86 100.000 100.000 97.208 90.655 81.598
87 100.000 100.000 97.302 90.864 81.905
88 100.000 100.000 97.396 91.062 82.187
89 100.000 100.000 97.466 91.228 82.440
90 100.000 100.000 97.536 91.384 82.669
91 100.000 100.000 97.607 91.539 82.900
92 100.000 100.000 97.677 91.716 83.181
93 100.000 100.000 97.748 91.893 83.464
94 100.000 100.000 97.818 92.059 83.721
95 100.000 100.000 97.888 92.226 83.975
96 100.000 100.000 97.959 92.376 84.191
97 100.000 100.000 98.029 92.531 84.420
98 100.000 100.000 98.099 92.681 84.637
99 100.000 100.000 98.146 92.794 84.814
100 100.000 100.000 98.193 92.912 85.005
101 100.000 100.000 98.240 93.067 85.284
102 100.000 100.000 98.287 93.169 85.434
103 100.000 100.000 98.334 93.293 85.635
104 100.000 100.000 98.381 93.394 85.787
105 100.000 100.000 98.428 93.523 86.001
106 100.000 100.000 98.475 93.636 86.179
107 100.000 100.000 98.522 93.738 86.329
108 100.000 100.000 98.569 93.866 86.547
109 100.000 100.000 98.616 93.995 86.761
110 100.000 100.000 98.663 94.108 86.939
111 100.000 100.000 98.710 94.209 87.089
112 100.000 100.000 98.756 94.333 87.292
113 100.000 100.000 98.803 94.435 87.442
114 100.000 100.000 98.850 94.536 87.592
115 100.000 100.000 98.897 94.649 87.769
116 100.000 100.000 98.944 94.778 87.987
117 100.000 100.000 98.991 94.896 88.178
118 100.000 100.000 99.038 94.998 88.328
119 100.000 100.000 99.085 95.105 88.492
120 100.000 100.000 99.132 95.228 88.694
121 100.000 100.000 99.179 95.330 88.844
122 100.000 100.000 99.226 95.448 89.034
123 100.000 100.000 99.273 95.630 89.369
124 100.000 100.000 99.320 95.732 89.519
125 100.000 100.000 99.343 95.786 89.602
126 100.000 100.000 99.366 95.839 89.685
127 100.000 100.000 99.390 95.914 89.824
128 100.000 100.000 99.413 96.006 90.001
129 100.000 100.000 99.437 96.081 90.140
130 100.000 100.000 99.460 96.145 90.253
131 100.000 100.000 99.484 96.204 90.351
132 100.000 100.000 99.507 96.268 90.462
133 100.000 100.000 99.531 96.354 90.624
134 100.000 100.000 99.554 96.413 90.722
135 100.000 100.000 99.578 96.483 90.848
136 100.000 100.000 99.601 96.552 90.974
137 100.000 100.000 99.625 96.606 91.057
138 100.000 100.000 99.648 96.665 91.155
139 100.000 100.000 99.672 96.735 91.290
140 100.000 100.000 99.695 96.794 91.388
141 100.000 100.000 99.718 96.869 91.531
142 100.000 100.000 99.742 96.922 91.614
143 100.000 100.000 99.765 96.981 91.712
144 100.000 100.000 99.789 97.046 91.823
145 100.000 100.000 99.812 97.121 91.965
146 100.000 100.000 99.836 97.180 92.063
147 100.000 100.000 99.859 97.239 92.161
148 100.000 100.000 99.883 97.298 92.259
149 100.000 100.000 99.906 97.351 92.342
150 100.000 100.000 99.930 97.405 92.426
151 100.000 100.000 99.953 97.459 92.509
152 100.000 100.000 99.977 97.523 92.622
153 100.000 100.000 100.000 97.577 92.705
154 100.000 100.000 100.000 97.582 92.723
155 100.000 100.000 100.000 97.587 92.744
156 100.000 100.000 100.000 97.614 92.818
157 100.000 100.000 100.000 97.641 92.892
158 100.000 100.000 100.000 97.684 93.003
159 100.000 100.000 100.000 97.705 93.062
160 100.000 100.000 100.000 97.716 93.093
161 100.000 100.000 100.000 97.721 93.112
162 100.000 100.000 100.000 97.743 93.173
163 100.000 100.000 100.000 97.764 93.232
164 100.000 100.000 100.000 97.807 93.345
165 100.000 100.000 100.000 97.818 93.376
166 100.000 100.000 100.000 97.829 93.408
167 100.000 100.000 100.000 97.850 93.467
168 100.000 100.000 100.000 97.866 93.511
169 100.000 100.000 100.000 97.877 93.543
170 100.000 100.000 100.000 97.888 93.574
171 100.000 100.000 100.000 97.914 93.648
172 100.000 100.000 100.000 97.920 93.666
173 100.000 100.000 100.000 97.925 93.683
174 100.000 100.000 100.000 97.957 93.770
175 100.000 100.000 100.000 97.968 93.801
176 100.000 100.000 100.000 97.989 93.861
177 100.000 100.000 100.000 98.022 93.948
178 100.000 100.000 100.000 98.064 94.058
179 100.000 100.000 100.000 98.113 94.186
180 100.000 100.000 100.000 98.118 94.205
181 100.000 100.000 100.000 98.150 94.291
182 100.000 100.000 100.000 98.161 94.323
183 100.000 100.000 100.000 98.193 94.406
184 100.000 100.000 100.000 98.209 94.452
185 100.000 100.000 100.000 98.231 94.515
186 100.000 100.000 100.000 98.241 94.550
187 100.000 100.000 100.000 98.247 94.567
188 100.000 100.000 100.000 98.257 94.598
189 100.000 100.000 100.000 98.279 94.658
190 100.000 100.000 100.000 98.295 94.704
191 100.000 100.000 100.000 98.300 94.722
192 100.000 100.000 100.000 98.311 94.754
193 100.000 100.000 100.000 98.327 94.800
194 100.000 100.000 100.000 98.343 94.848
195 100.000 100.000 100.000 98.343 94.855
196 100.000 100.000 100.000 98.365 94.915
197 100.000 100.000 100.000 98.413 95.042
198 100.000 100.000 100.000 98.434 95.101
199 100.000 100.000 100.000 98.440 95.118
200 100.000 100.000 100.000 98.445 95.137
201 100.000 100.000 100.000 98.450 95.153
202 100.000 100.000 100.000 98.456 95.172
203 100.000 100.000 100.000 98.467 95.207
204 100.000 100.000 100.000 98.488 95.266
205 100.000 100.000 100.000 98.520 95.353
206 100.000 100.000 100.000 98.526 95.371
207 100.000 100.000 100.000 98.531 95.388
208 100.000 100.000 100.000 98.536 95.407
209 100.000 100.000 100.000 98.552 95.453
210 100.000 100.000 100.000 98.563 95.488
211 100.000 100.000 100.000 98.574 95.519
212 100.000 100.000 100.000 98.585 95.554
213 100.000 100.000 100.000 98.590 95.571
214 100.000 100.000 100.000 98.606 95.616
215 100.000 100.000 100.000 98.611 95.632
216 100.000 100.000 100.000 98.638 95.704
217 100.000 100.000 100.000 98.660 95.767
218 100.000 100.000 100.000 98.670 95.799
219 100.000 100.000 100.000 98.676 95.815
220 100.000 100.000 100.000 98.676 95.823
221 100.000 100.000 100.000 98.697 95.885
222 100.000 100.000 100.000 98.708 95.919
223 100.000 100.000 100.000 98.724 95.963
224 100.000 100.000 100.000 98.729 95.980
225 100.000 100.000 100.000 98.740 96.011
226 100.000 100.000 100.000 98.740 96.013
227 100.000 100.000 100.000 98.745 96.032
228 100.000 100.000 100.000 98.745 96.033
229 100.000 100.000 100.000 98.756 96.065
230 100.000 100.000 100.000 98.761 96.082
231 100.000 100.000 100.000 98.767 96.102
232 100.000 100.000 100.000 98.778 96.137
233 100.000 100.000 100.000 98.788 96.172
234 100.000 100.000 100.000 98.788 96.176
235 100.000 100.000 100.000 98.799 96.207
236 100.000 100.000 100.000 98.810 96.239
237 100.000 100.000 100.000 98.815 96.255
238 100.000 100.000 100.000 98.820 96.272
239 100.000 100.000 100.000 98.826 96.289
240 100.000 100.000 100.000 98.831 96.305
241 100.000 100.000 100.000 98.869 96.401
242 100.000 100.000 100.000 98.879 96.433
243 100.000 100.000 100.000 98.944 96.596
244 100.000 100.000 100.000 98.954 96.629
245 100.000 100.000 100.000 98.965 96.660
246 100.000 100.000 100.000 98.981 96.707
247 100.000 100.000 100.000 98.987 96.723
248 100.000 100.000 100.000 98.997 96.758
249 100.000 100.000 100.000 99.003 96.775
250 100.000 100.000 100.000 99.013 96.808
251 100.000 100.000 100.000 99.024 96.843
252 100.000 100.000 100.000 99.024 96.845
253 100.000 100.000 100.000 99.035 96.877
254 100.000 100.000 100.000 99.040 96.893
255 100.000 100.000 100.000 99.046 96.910
256 100.000 100.000 100.000 99.051 96.927
257 100.000 100.000 100.000 99.072 96.986
258 100.000 100.000 100.000 99.083 97.017
259 100.000 100.000 100.000 99.083 97.019
260 100.000 100.000 100.000 99.099 97.069
261 100.000 100.000 100.000 99.105 97.086
262 100.000 100.000 100.000 99.126 97.149
263 100.000 100.000 100.000 99.137 97.180
264 100.000 100.000 100.000 99.147 97.211
265 100.000 100.000 100.000 99.153 97.228
266 100.000 100.000 100.000 99.153 97.232
267 100.000 100.000 100.000 99.158 97.248
268 100.000 100.000 100.000 99.164 97.265
269 100.000 100.000 100.000 99.174 97.296
270 100.000 100.000 100.000 99.180 97.313
271 100.000 100.000 100.000 99.180 97.315
272 100.000 100.000 100.000 99.190 97.346
273 100.000 100.000 100.000 99.196 97.365
274 100.000 100.000 100.000 99.196 97.367
275 100.000 100.000 100.000 99.217 97.426
276 100.000 100.000 100.000 99.228 97.457
277 100.000 100.000 100.000 99.233 97.476
278 100.000 100.000 100.000 99.255 97.535
279 100.000 100.000 100.000 99.260 97.552
280 100.000 100.000 100.000 99.271 97.583
281 100.000 100.000 100.000 99.276 97.600
282 100.000 100.000 100.000 99.287 97.631
283 100.000 100.000 100.000 99.292 97.653
284 100.000 100.000 100.000 99.357 97.816
285 100.000 100.000 100.000 99.378 97.875
286 100.000 100.000 100.000 99.378 97.879
287 100.000 100.000 100.000 99.389 97.910
288 100.000 100.000 100.000 99.399 97.942
289 100.000 100.000 100.000 99.416 97.986
290 100.000 100.000 100.000 99.421 98.003
291 100.000 100.000 100.000 99.421 98.005
292 100.000 100.000 100.000 99.421 98.007
293 100.000 100.000 100.000 99.426 98.023
294 100.000 100.000 100.000 99.437 98.062
295 100.000 100.000 100.000 99.448 98.093
296 100.000 100.000 100.000 99.448 98.101
297 100.000 100.000 100.000 99.453 98.117
298 100.000 100.000 100.000 99.453 98.119
299 100.000 100.000 100.000 99.464 98.151
300 100.000 100.000 100.000 99.464 98.154
301 100.000 100.000 100.000 99.469 98.171
302 100.000 100.000 100.000 99.480 98.203
303 100.000 100.000 100.000 99.501 98.265
304 100.000 100.000 100.000 99.507 98.282
305 100.000 100.000 100.000 99.507 98.286
306 100.000 100.000 100.000 99.512 98.302
307 100.000 100.000 100.000 99.517 98.319
308 100.000 100.000 100.000 99.517 98.321
309 100.000 100.000 100.000 99.528 98.352
310 100.000 100.000 100.000 99.539 98.384
311 100.000 100.000 100.000 99.539 98.387
312 100.000 100.000 100.000 99.544 98.408
313 100.000 100.000 100.000 99.566 98.467
314 100.000 100.000 100.000 99.576 98.498
315 100.000 100.000 100.000 99.587 98.530
316 100.000 100.000 100.000 99.593 98.547
317 100.000 100.000 100.000 99.603 98.578
318 100.000 100.000 100.000 99.603 98.582
319 100.000 100.000 100.000 99.609 98.598
320 100.000 100.000 100.000 99.614 98.615
321 100.000 100.000 100.000 99.614 98.619
322 100.000 100.000 100.000 99.619 98.635
323 100.000 100.000 100.000 99.625 98.652
324 100.000 100.000 100.000 99.625 98.656
325 100.000 100.000 100.000 99.635 98.687
326 100.000 100.000 100.000 99.641 98.704
327 100.000 100.000 100.000 99.646 98.720
328 100.000 100.000 100.000 99.646 98.724
329 100.000 100.000 100.000 99.646 98.726
330 100.000 100.000 100.000 99.657 98.757
331 100.000 100.000 100.000 99.662 98.778
332 100.000 100.000 100.000 99.668 98.794
333 100.000 100.000 100.000 99.673 98.811
334 100.000 100.000 100.000 99.673 98.813
335 100.000 100.000 100.000 99.678 98.829
336 100.000 100.000 100.000 99.689 98.861
337 100.000 100.000 100.000 99.694 98.878
338 100.000 100.000 100.000 99.700 98.894
339 100.000 100.000 100.000 99.705 98.911
340 100.000 100.000 100.000 99.710 98.927
341 100.000 100.000 100.000 99.710 98.929
342 100.000 100.000 100.000 99.710 98.933
343 100.000 100.000 100.000 99.721 98.968
344 100.000 100.000 100.000 99.732 99.000
345 100.000 100.000 100.000 99.732 99.003
346 100.000 100.000 100.000 99.732 99.005
347 100.000 100.000 100.000 99.748 99.051
348 100.000 100.000 100.000 99.759 99.083
349 100.000 100.000 100.000 99.764 99.099
350 100.000 100.000 100.000 99.764 99.101
351 100.000 100.000 100.000 99.775 99.133
352 100.000 100.000 100.000 99.775 99.140
353 100.000 100.000 100.000 99.775 99.144
354 100.000 100.000 100.000 99.775 99.148
355 100.000 100.000 100.000 99.780 99.164
356 100.000 100.000 100.000 99.786 99.181
357 100.000 100.000 100.000 99.786 99.184
358 100.000 100.000 100.000 99.786 99.188
359 100.000 100.000 100.000 99.786 99.190
360 100.000 100.000 100.000 99.786 99.194
361 100.000 100.000 100.000 99.791 99.210
362 100.000 100.000 100.000 99.791 99.212
363 100.000 100.000 100.000 99.802 99.244
364 100.000 100.000 100.000 99.807 99.260
365 100.000 100.000 100.000 99.812 99.277
366 100.000 100.000 100.000 99.818 99.295
367 100.000 100.000 100.000 99.828 99.327
368 100.000 100.000 100.000 99.828 99.329
369 100.000 100.000 100.000 99.834 99.345
370 100.000 100.000 100.000 99.834 99.347
371 100.000 100.000 100.000 99.834 99.351
372 100.000 100.000 100.000 99.834 99.353
373 100.000 100.000 100.000 99.850 99.399
374 100.000 100.000 100.000 99.850 99.403
375 100.000 100.000 100.000 99.850 99.406
376 100.000 100.000 100.000 99.850 99.412
377 100.000 100.000 100.000 99.855 99.429
378 100.000 100.000 100.000 99.861 99.445
379 100.000 100.000 100.000 99.866 99.462
380 100.000 100.000 100.000 99.871 99.479
381 100.000 100.000 100.000 99.871 99.482
382 100.000 100.000 100.000 99.882 99.514
383 100.000 100.000 100.000 99.882 99.517
384 100.000 100.000 100.000 99.882 99.525
385 100.000 100.000 100.000 99.893 99.556
386 100.000 100.000 100.000 99.893 99.558
387 100.000 100.000 100.000 99.903 99.593
388 100.000 100.000 100.000 99.903 99.597
389 100.000 100.000 100.000 99.903 99.601
390 100.000 100.000 100.000 99.903 99.604
391 100.000 100.000 100.000 99.909 99.621
392 100.000 100.000 100.000 99.909 99.623
393 100.000 100.000 100.000 99.909 99.625
394 100.000 100.000 100.000 99.914 99.641
395 100.000 100.000 100.000 99.925 99.673
396 100.000 100.000 100.000 99.925 99.676
397 100.000 100.000 100.000 99.930 99.693
398 100.000 100.000 100.000 99.941 99.724
399 100.000 100.000 100.000 99.946 99.741
400 100.000 100.000 100.000 99.946 99.745
401 100.000 100.000 100.000 99.946 99.749
402 100.000 100.000 100.000 99.952 99.765
403 100.000 100.000 100.000 99.952 99.769
404 100.000 100.000 100.000 99.952 99.771
405 100.000 100.000 100.000 99.952 99.782
406 100.000 100.000 100.000 99.952 99.784
407 100.000 100.000 100.000 99.952 99.787
408 100.000 100.000 100.000 99.952 99.791
409 100.000 100.000 100.000 99.952 99.795
410 100.000 100.000 100.000 99.952 99.798
411 100.000 100.000 100.000 99.957 99.815
412 100.000 100.000 100.000 99.962 99.832
413 100.000 100.000 100.000 99.962 99.839
414 100.000 100.000 100.000 99.962 99.843
415 100.000 100.000 100.000 99.968 99.859
416 100.000 100.000 100.000 99.973 99.876
417 100.000 100.000 100.000 99.973 99.880
418 100.000 100.000 100.000 99.979 99.896
419 100.000 100.000 100.000 99.979 99.900
420 100.000 100.000 100.000 99.979 99.904
421 100.000 100.000 100.000 99.979 99.906
422 100.000 100.000 100.000 99.979 99.908
423 100.000 100.000 100.000 99.979 99.909
424 100.000 100.000 100.000 99.979 99.911
425 100.000 100.000 100.000 99.984 99.928
426 100.000 100.000 100.000 99.984 99.932
427 100.000 100.000 100.000 99.995 99.963
428 100.000 100.000 100.000 99.995 99.967
429 100.000 100.000 100.000 99.995 99.970
430 100.000 100.000 100.000 99.995 99.974
431 100.000 100.000 100.000 99.995 99.976
432 100.000 100.000 100.000 100.000 99.993
433 100.000 100.000 100.000 100.000 99.994
434 100.000 100.000 100.000 100.000 99.996
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 47.619 23.858 12.952 7.587 4.749
3 57.143 31.641 18.254 11.072 7.053
4 64.286 38.579 23.322 14.503 9.348
5 73.810 46.024 28.414 17.903 11.626
6 83.333 53.807 33.740 21.409 13.941
7 88.095 59.222 38.081 24.546 16.134
8 90.476 63.113 41.764 27.457 18.267
9 95.238 68.697 46.222 30.647 20.478
10 100.000 73.435 50.000 33.457 22.514
11 100.000 75.465 52.557 35.784 24.384
12 100.000 76.988 54.716 37.896 26.161
13 100.000 79.019 57.227 40.164 27.984
14 100.000 80.372 59.338 42.282 29.767
15 100.000 82.234 61.614 44.368 31.487
16 100.000 83.249 63.210 46.078 33.038
17 100.000 84.772 65.204 47.992 34.667
18 100.000 86.125 67.058 49.810 36.230
19 100.000 87.310 68.700 51.466 37.702
20 100.000 88.325 70.155 52.994 39.107
21 100.000 89.171 71.422 54.356 40.383
22 100.000 89.679 72.407 55.568 41.602
23 100.000 90.355 73.534 56.849 42.841
24 100.000 91.201 74.847 58.238 44.098
25 100.000 91.878 75.950 59.477 45.289
26 100.000 92.555 77.053 60.715 46.480
27 100.000 92.893 77.851 61.771 47.590
28 100.000 93.401 78.719 62.822 48.664
29 100.000 93.909 79.634 63.911 49.751
30 100.000 94.416 80.526 64.967 50.811
31 100.000 94.755 81.276 65.937 51.822
32 100.000 95.262 82.145 66.956 52.836
33 100.000 95.431 82.708 67.798 53.784
34 100.000 95.601 83.294 68.656 54.724
35 100.000 95.939 83.975 69.535 55.665
36 100.000 96.108 84.514 70.334 56.566
37 100.000 96.616 85.312 71.256 57.501
38 100.000 96.954 85.946 72.050 58.339
39 100.000 97.293 86.579 72.838 59.166
40 100.000 97.462 87.001 73.471 59.911
41 100.000 97.631 87.447 74.125 60.669
42 100.000 97.800 87.893 74.768 61.403
43 100.000 97.970 88.339 75.412 62.137
44 100.000 97.970 88.620 75.958 62.840
45 100.000 98.139 89.066 76.602 63.574
46 100.000 98.308 89.465 77.192 64.277
47 100.000 98.308 89.747 77.722 64.946
48 100.000 98.477 90.145 78.291 65.599
49 100.000 98.646 90.544 78.854 66.244
50 100.000 98.816 90.896 79.358 66.845
51 100.000 98.816 91.107 79.781 67.409
52 100.000 99.154 91.600 80.350 68.012
53 100.000 99.323 91.976 80.864 68.593
54 100.000 99.323 92.187 81.283 69.138
55 100.000 99.323 92.374 81.669 69.664
56 100.000 99.323 92.562 82.055 70.189
57 100.000 99.492 92.914 82.532 70.731
58 100.000 99.492 93.102 82.907 71.232
59 100.000 99.492 93.290 83.272 71.709
60 100.000 99.492 93.454 83.615 72.182
61 100.000 99.662 93.782 84.054 72.678
62 100.000 99.662 93.947 84.392 73.140
63 100.000 99.831 94.252 84.800 73.615
64 100.000 99.831 94.416 85.132 74.067
65 100.000 99.831 94.580 85.465 74.518
66 100.000 99.831 94.721 85.770 74.954
67 100.000 99.831 94.838 86.054 75.385
68 100.000 99.831 94.979 86.355 75.809
69 100.000 99.831 95.120 86.650 76.221
70 100.000 99.831 95.260 86.944 76.631
71 100.000 99.831 95.401 87.239 77.042
72 100.000 99.831 95.542 87.534 77.456
73 100.000 99.831 95.683 87.824 77.856
74 100.000 99.831 95.800 88.092 78.251
75 100.000 99.831 95.941 88.376 78.638
76 100.000 99.831 96.082 88.660 79.017
77 100.000 99.831 96.222 88.939 79.389
78 100.000 99.831 96.363 89.212 79.747
79 100.000 99.831 96.481 89.459 80.095
80 100.000 99.831 96.574 89.684 80.439
81 100.000 99.831 96.692 89.931 80.787
82 100.000 99.831 96.786 90.151 81.118
83 100.000 99.831 96.879 90.360 81.426
84 100.000 99.831 96.973 90.569 81.735
85 100.000 99.831 97.020 90.735 82.037
86 100.000 99.831 97.067 90.901 82.334
87 100.000 99.831 97.137 91.084 82.630
88 100.000 99.831 97.184 91.244 82.921
89 100.000 100.000 97.443 91.539 83.226
90 100.000 100.000 97.489 91.695 83.503
91 100.000 100.000 97.536 91.850 83.781
92 100.000 100.000 97.630 92.049 84.060
93 100.000 100.000 97.724 92.242 84.328
94 100.000 100.000 97.818 92.435 84.596
95 100.000 100.000 97.888 92.606 84.862
96 100.000 100.000 97.959 92.773 85.119
97 100.000 100.000 98.029 92.939 85.376
98 100.000 100.000 98.099 93.100 85.621
99 100.000 100.000 98.170 93.260 85.866
100 100.000 100.000 98.217 93.400 86.109
101 100.000 100.000 98.287 93.561 86.355
102 100.000 100.000 98.334 93.695 86.584
103 100.000 100.000 98.405 93.845 86.800
104 100.000 100.000 98.475 93.995 87.017
105 100.000 100.000 98.522 94.124 87.231
106 100.000 100.000 98.569 94.247 87.436
107 100.000 100.000 98.616 94.370 87.642
108 100.000 100.000 98.663 94.494 87.849
109 100.000 100.000 98.710 94.612 88.039
110 100.000 100.000 98.756 94.730 88.230
111 100.000 100.000 98.803 94.842 88.411
112 100.000 100.000 98.850 94.955 88.588
113 100.000 100.000 98.897 95.067 88.770
114 100.000 100.000 98.944 95.180 88.949
115 100.000 100.000 98.944 95.244 89.104
116 100.000 100.000 98.991 95.351 89.269
117 100.000 100.000 99.038 95.459 89.434
118 100.000 100.000 99.061 95.544 89.600
119 100.000 100.000 99.061 95.609 89.763
120 100.000 100.000 99.108 95.716 89.931
121 100.000 100.000 99.155 95.818 90.081
122 100.000 100.000 99.179 95.898 90.234
123 100.000 100.000 99.226 96.000 90.384
124 100.000 100.000 99.249 96.081 90.538
125 100.000 100.000 99.296 96.183 90.687
126 100.000 100.000 99.343 96.284 90.837
127 100.000 100.000 99.390 96.386 90.987
128 100.000 100.000 99.413 96.461 91.126
129 100.000 100.000 99.437 96.536 91.266
130 100.000 100.000 99.437 96.590 91.405
131 100.000 100.000 99.460 96.665 91.545
132 100.000 100.000 99.484 96.735 91.671
133 100.000 100.000 99.507 96.804 91.797
134 100.000 100.000 99.531 96.869 91.908
135 100.000 100.000 99.531 96.912 92.022
136 100.000 100.000 99.554 96.976 92.135
137 100.000 100.000 99.554 97.019 92.246
138 100.000 100.000 99.578 97.083 92.357
139 100.000 100.000 99.601 97.148 92.468
140 100.000 100.000 99.625 97.212 92.581
141 100.000 100.000 99.648 97.276 92.696
142 100.000 100.000 99.672 97.341 92.807
143 100.000 100.000 99.695 97.405 92.918
144 100.000 100.000 99.718 97.469 93.028
145 100.000 100.000 99.742 97.528 93.126
146 100.000 100.000 99.765 97.587 93.224
147 100.000 100.000 99.789 97.646 93.322
148 100.000 100.000 99.812 97.705 93.422
149 100.000 100.000 99.812 97.743 93.522
150 100.000 100.000 99.836 97.802 93.620
151 100.000 100.000 99.836 97.839 93.720
152 100.000 100.000 99.859 97.898 93.818
153 100.000 100.000 99.883 97.957 93.918
154 100.000 100.000 99.883 97.995 94.014
155 100.000 100.000 99.906 98.054 94.112
156 100.000 100.000 99.906 98.086 94.199
157 100.000 100.000 99.906 98.118 94.286
158 100.000 100.000 99.930 98.172 94.369
159 100.000 100.000 99.953 98.225 94.452
160 100.000 100.000 99.953 98.257 94.539
161 100.000 100.000 99.953 98.290 94.626
162 100.000 100.000 99.977 98.343 94.709
163 100.000 100.000 99.977 98.375 94.796
164 100.000 100.000 100.000 98.429 94.880
165 100.000 100.000 100.000 98.461 94.966
166 100.000 100.000 100.000 98.493 95.053
167 100.000 100.000 100.000 98.526 95.140
168 100.000 100.000 100.000 98.552 95.216
169 100.000 100.000 100.000 98.579 95.288
170 100.000 100.000 100.000 98.606 95.368
171 100.000 100.000 100.000 98.633 95.440
172 100.000 100.000 100.000 98.660 95.512
173 100.000 100.000 100.000 98.686 95.588
174 100.000 100.000 100.000 98.713 95.664
175 100.000 100.000 100.000 98.735 95.723
176 100.000 100.000 100.000 98.756 95.793
177 100.000 100.000 100.000 98.778 95.860
178 100.000 100.000 100.000 98.799 95.922
179 100.000 100.000 100.000 98.820 95.984
180 100.000 100.000 100.000 98.842 96.045
181 100.000 100.000 100.000 98.863 96.104
182 100.000 100.000 100.000 98.885 96.167
183 100.000 100.000 100.000 98.906 96.226
184 100.000 100.000 100.000 98.928 96.289
185 100.000 100.000 100.000 98.949 96.348
186 100.000 100.000 100.000 98.971 96.411
187 100.000 100.000 100.000 98.992 96.474
188 100.000 100.000 100.000 99.013 96.533
189 100.000 100.000 100.000 99.035 96.592
190 100.000 100.000 100.000 99.056 96.655
191 100.000 100.000 100.000 99.078 96.714
192 100.000 100.000 100.000 99.099 96.777
193 100.000 100.000 100.000 99.121 96.836
194 100.000 100.000 100.000 99.142 96.895
195 100.000 100.000 100.000 99.164 96.958
196 100.000 100.000 100.000 99.185 97.021
197 100.000 100.000 100.000 99.201 97.067
198 100.000 100.000 100.000 99.217 97.113
199 100.000 100.000 100.000 99.233 97.163
200 100.000 100.000 100.000 99.249 97.213
201 100.000 100.000 100.000 99.265 97.259
202 100.000 100.000 100.000 99.282 97.306
203 100.000 100.000 100.000 99.298 97.354
204 100.000 100.000 100.000 99.314 97.400
205 100.000 100.000 100.000 99.330 97.444
206 100.000 100.000 100.000 99.346 97.491
207 100.000 100.000 100.000 99.362 97.537
208 100.000 100.000 100.000 99.373 97.570
209 100.000 100.000 100.000 99.383 97.603
210 100.000 100.000 100.000 99.394 97.635
211 100.000 100.000 100.000 99.405 97.666
212 100.000 100.000 100.000 99.416 97.698
213 100.000 100.000 100.000 99.426 97.729
214 100.000 100.000 100.000 99.437 97.762
215 100.000 100.000 100.000 99.448 97.794
216 100.000 100.000 100.000 99.458 97.825
217 100.000 100.000 100.000 99.469 97.859
218 100.000 100.000 100.000 99.480 97.890
219 100.000 100.000 100.000 99.491 97.925
220 100.000 100.000 100.000 99.501 97.958
221 100.000 100.000 100.000 99.512 97.994
222 100.000 100.000 100.000 99.523 98.025
223 100.000 100.000 100.000 99.534 98.056
224 100.000 100.000 100.000 99.544 98.090
225 100.000 100.000 100.000 99.555 98.125
226 100.000 100.000 100.000 99.566 98.156
227 100.000 100.000 100.000 99.576 98.188
228 100.000 100.000 100.000 99.587 98.223
229 100.000 100.000 100.000 99.598 98.254
230 100.000 100.000 100.000 99.609 98.288
231 100.000 100.000 100.000 99.619 98.325
232 100.000 100.000 100.000 99.630 98.362
233 100.000 100.000 100.000 99.641 98.393
234 100.000 100.000 100.000 99.651 98.424
235 100.000 100.000 100.000 99.662 98.458
236 100.000 100.000 100.000 99.673 98.489
237 100.000 100.000 100.000 99.684 98.521
238 100.000 100.000 100.000 99.694 98.552
239 100.000 100.000 100.000 99.705 98.584
240 100.000 100.000 100.000 99.716 98.615
241 100.000 100.000 100.000 99.721 98.632
242 100.000 100.000 100.000 99.727 98.648
243 100.000 100.000 100.000 99.732 98.665
244 100.000 100.000 100.000 99.737 98.683
245 100.000 100.000 100.000 99.743 98.700
246 100.000 100.000 100.000 99.748 98.717
247 100.000 100.000 100.000 99.753 98.733
248 100.000 100.000 100.000 99.759 98.750
249 100.000 100.000 100.000 99.764 98.767
250 100.000 100.000 100.000 99.769 98.783
251 100.000 100.000 100.000 99.775 98.802
252 100.000 100.000 100.000 99.780 98.818
253 100.000 100.000 100.000 99.786 98.835
254 100.000 100.000 100.000 99.791 98.852
255 100.000 100.000 100.000 99.796 98.868
256 100.000 100.000 100.000 99.802 98.885
257 100.000 100.000 100.000 99.807 98.902
258 100.000 100.000 100.000 99.812 98.918
259 100.000 100.000 100.000 99.818 98.937
260 100.000 100.000 100.000 99.823 98.953
261 100.000 100.000 100.000 99.828 98.970
262 100.000 100.000 100.000 99.834 98.987
263 100.000 100.000 100.000 99.839 99.003
264 100.000 100.000 100.000 99.845 99.020
265 100.000 100.000 100.000 99.850 99.037
266 100.000 100.000 100.000 99.855 99.053
267 100.000 100.000 100.000 99.861 99.070
268 100.000 100.000 100.000 99.866 99.088
269 100.000 100.000 100.000 99.871 99.107
270 100.000 100.000 100.000 99.877 99.123
271 100.000 100.000 100.000 99.882 99.144
272 100.000 100.000 100.000 99.887 99.160
273 100.000 100.000 100.000 99.893 99.177
274 100.000 100.000 100.000 99.898 99.196
275 100.000 100.000 100.000 99.903 99.216
276 100.000 100.000 100.000 99.909 99.233
277 100.000 100.000 100.000 99.914 99.249
278 100.000 100.000 100.000 99.920 99.266
279 100.000 100.000 100.000 99.925 99.283
280 100.000 100.000 100.000 99.930 99.299
281 100.000 100.000 100.000 99.936 99.316
282 100.000 100.000 100.000 99.941 99.332
283 100.000 100.000 100.000 99.946 99.351
284 100.000 100.000 100.000 99.952 99.368
285 100.000 100.000 100.000 99.957 99.384
286 100.000 100.000 100.000 99.962 99.405
287 100.000 100.000 100.000 99.968 99.421
288 100.000 100.000 100.000 99.973 99.438
289 100.000 100.000 100.000 99.979 99.454
290 100.000 100.000 100.000 99.984 99.471
291 100.000 100.000 100.000 99.989 99.488
292 100.000 100.000 100.000 99.995 99.504
293 100.000 100.000 100.000 100.000 99.521
294 100.000 100.000 100.000 100.000 99.525
295 100.000 100.000 100.000 100.000 99.532
296 100.000 100.000 100.000 100.000 99.536
297 100.000 100.000 100.000 100.000 99.538
298 100.000 100.000 100.000 100.000 99.540
299 100.000 100.000 100.000 100.000 99.541
300 100.000 100.000 100.000 100.000 99.543
301 100.000 100.000 100.000 100.000 99.545
302 100.000 100.000 100.000 100.000 99.547
303 100.000 100.000 100.000 100.000 99.549
304 100.000 100.000 100.000 100.000 99.551
305 100.000 100.000 100.000 100.000 99.554
306 100.000 100.000 100.000 100.000 99.562
307 100.000 100.000 100.000 100.000 99.564
308 100.000 100.000 100.000 100.000 99.567
309 100.000 100.000 100.000 100.000 99.571
310 100.000 100.000 100.000 100.000 99.575
311 100.000 100.000 100.000 100.000 99.578
312 100.000 100.000 100.000 100.000 99.582
313 100.000 100.000 100.000 100.000 99.586
314 100.000 100.000 100.000 100.000 99.589
315 100.000 100.000 100.000 100.000 99.593
316 100.000 100.000 100.000 100.000 99.597
317 100.000 100.000 100.000 100.000 99.601
318 100.000 100.000 100.000 100.000 99.602
319 100.000 100.000 100.000 100.000 99.606
320 100.000 100.000 100.000 100.000 99.610
321 100.000 100.000 100.000 100.000 99.612
322 100.000 100.000 100.000 100.000 99.615
323 100.000 100.000 100.000 100.000 99.623
324 100.000 100.000 100.000 100.000 99.625
325 100.000 100.000 100.000 100.000 99.626
326 100.000 100.000 100.000 100.000 99.630
327 100.000 100.000 100.000 100.000 99.632
328 100.000 100.000 100.000 100.000 99.634
329 100.000 100.000 100.000 100.000 99.641
330 100.000 100.000 100.000 100.000 99.643
331 100.000 100.000 100.000 100.000 99.647
332 100.000 100.000 100.000 100.000 99.649
333 100.000 100.000 100.000 100.000 99.652
334 100.000 100.000 100.000 100.000 99.654
335 100.000 100.000 100.000 100.000 99.658
336 100.000 100.000 100.000 100.000 99.660
337 100.000 100.000 100.000 100.000 99.662
338 100.000 100.000 100.000 100.000 99.665
339 100.000 100.000 100.000 100.000 99.671
340 100.000 100.000 100.000 100.000 99.678
341 100.000 100.000 100.000 100.000 99.680
342 100.000 100.000 100.000 100.000 99.682
343 100.000 100.000 100.000 100.000 99.686
344 100.000 100.000 100.000 100.000 99.687
345 100.000 100.000 100.000 100.000 99.691
346 100.000 100.000 100.000 100.000 99.695
347 100.000 100.000 100.000 100.000 99.699
348 100.000 100.000 100.000 100.000 99.700
349 100.000 100.000 100.000 100.000 99.704
350 100.000 100.000 100.000 100.000 99.708
351 100.000 100.000 100.000 100.000 99.712
352 100.000 100.000 100.000 100.000 99.713
353 100.000 100.000 100.000 100.000 99.715
354 100.000 100.000 100.000 100.000 99.723
355 100.000 100.000 100.000 100.000 99.724
356 100.000 100.000 100.000 100.000 99.726
357 100.000 100.000 100.000 100.000 99.728
358 100.000 100.000 100.000 100.000 99.730
359 100.000 100.000 100.000 100.000 99.734
360 100.000 100.000 100.000 100.000 99.737
361 100.000 100.000 100.000 100.000 99.745
362 100.000 100.000 100.000 100.000 99.749
363 100.000 100.000 100.000 100.000 99.752
364 100.000 100.000 100.000 100.000 99.754
365 100.000 100.000 100.000 100.000 99.756
366 100.000 100.000 100.000 100.000 99.760
367 100.000 100.000 100.000 100.000 99.761
368 100.000 100.000 100.000 100.000 99.776
369 100.000 100.000 100.000 100.000 99.778
370 100.000 100.000 100.000 100.000 99.782
371 100.000 100.000 100.000 100.000 99.789
372 100.000 100.000 100.000 100.000 99.793
373 100.000 100.000 100.000 100.000 99.797
374 100.000 100.000 100.000 100.000 99.800
375 100.000 100.000 100.000 100.000 99.802
376 100.000 100.000 100.000 100.000 99.806
377 100.000 100.000 100.000 100.000 99.810
378 100.000 100.000 100.000 100.000 99.813
379 100.000 100.000 100.000 100.000 99.817
380 100.000 100.000 100.000 100.000 99.821
381 100.000 100.000 100.000 100.000 99.822
382 100.000 100.000 100.000 100.000 99.826
383 100.000 100.000 100.000 100.000 99.830
384 100.000 100.000 100.000 100.000 99.835
385 100.000 100.000 100.000 100.000 99.837
386 100.000 100.000 100.000 100.000 99.841
387 100.000 100.000 100.000 100.000 99.843
388 100.000 100.000 100.000 100.000 99.847
389 100.000 100.000 100.000 100.000 99.850
390 100.000 100.000 100.000 100.000 99.852
391 100.000 100.000 100.000 100.000 99.854
392 100.000 100.000 100.000 100.000 99.858
393 100.000 100.000 100.000 100.000 99.861
394 100.000 100.000 100.000 100.000 99.863
395 100.000 100.000 100.000 100.000 99.865
396 100.000 100.000 100.000 100.000 99.869
397 100.000 100.000 100.000 100.000 99.872
398 100.000 100.000 100.000 100.000 99.874
399 100.000 100.000 100.000 100.000 99.876
400 100.000 100.000 100.000 100.000 99.880
401 100.000 100.000 100.000 100.000 99.883
402 100.000 100.000 100.000 100.000 99.887
403 100.000 100.000 100.000 100.000 99.889
404 100.000 100.000 100.000 100.000 99.893
405 100.000 100.000 100.000 100.000 99.895
406 100.000 100.000 100.000 100.000 99.896
407 100.000 100.000 100.000 100.000 99.900
408 100.000 100.000 100.000 100.000 99.908
409 100.000 100.000 100.000 100.000 99.911
410 100.000 100.000 100.000 100.000 99.913
411 100.000 100.000 100.000 100.000 99.917
412 100.000 100.000 100.000 100.000 99.920
413 100.000 100.000 100.000 100.000 99.922
414 100.000 100.000 100.000 100.000 99.924
415 100.000 100.000 100.000 100.000 99.928
416 100.000 100.000 100.000 100.000 99.930
417 100.000 100.000 100.000 100.000 99.932
418 100.000 100.000 100.000 100.000 99.935
419 100.000 100.000 100.000 100.000 99.937
420 100.000 100.000 100.000 100.000 99.941
421 100.000 100.000 100.000 100.000 99.945
422 100.000 100.000 100.000 100.000 99.946
423 100.000 100.000 100.000 100.000 99.948
424 100.000 100.000 100.000 100.000 99.952
425 100.000 100.000 100.000 100.000 99.956
426 100.000 100.000 100.000 100.000 99.967
427 100.000 100.000 100.000 100.000 99.970
428 100.000 100.000 100.000 100.000 99.972
429 100.000 100.000 100.000 100.000 99.976
430 100.000 100.000 100.000 100.000 99.982
431 100.000 100.000 100.000 100.000 99.983
432 100.000 100.000 100.000 100.000 99.987
433 100.000 100.000 100.000 100.000 99.991
434 100.000 100.000 100.000 100.000 99.994
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

grep

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 30.952 13.198 6.710 3.834 2.380
2 42.857 21.658 12.107 7.292 4.656
3 54.762 30.288 17.644 10.831 6.970
4 61.905 36.717 22.337 14.074 9.191
5 69.048 42.978 26.889 17.238 11.375
6 73.810 48.562 31.347 20.428 13.586
7 80.952 54.992 36.039 23.672 15.807
8 80.952 57.360 38.878 26.165 17.760
9 80.952 59.729 41.671 28.610 19.683
10 83.333 62.944 44.721 31.124 21.617
11 85.714 66.497 48.076 33.816 23.629
12 90.476 71.066 51.689 36.518 25.610
13 92.857 73.773 54.247 38.711 27.376
14 92.857 75.635 56.546 40.813 29.097
15 95.238 78.173 58.916 42.866 30.780
16 95.238 79.357 60.605 44.587 32.308
17 95.238 80.880 62.529 46.405 33.854
18 100.000 85.110 65.791 48.845 35.666
19 100.000 85.787 67.034 50.276 37.025
20 100.000 86.802 68.536 51.842 38.440
21 100.000 87.648 69.873 53.295 39.790
22 100.000 88.663 71.305 54.775 41.139
23 100.000 89.679 72.877 56.421 42.606
24 100.000 90.355 73.956 57.648 43.808
25 100.000 91.371 75.411 59.144 45.156
26 100.000 91.878 76.396 60.335 46.338
27 100.000 92.386 77.311 61.434 47.449
28 100.000 93.232 78.555 62.753 48.677
29 100.000 93.739 79.423 63.787 49.729
30 100.000 93.909 80.033 64.683 50.715
31 100.000 94.247 80.760 65.621 51.699
32 100.000 94.247 81.206 66.399 52.605
33 100.000 94.416 81.793 67.251 53.538
34 100.000 94.585 82.309 68.012 54.400
35 100.000 95.431 83.482 69.213 55.499
36 100.000 95.939 84.280 70.136 56.434
37 100.000 96.277 84.913 70.945 57.305
38 100.000 96.785 85.758 71.921 58.274
39 100.000 97.124 86.368 72.693 59.108
40 100.000 97.462 86.978 73.455 59.920
41 100.000 97.462 87.236 73.964 60.586
42 100.000 97.631 87.705 74.639 61.348
43 100.000 97.631 87.940 75.106 61.967
44 100.000 97.800 88.386 75.749 62.701
45 100.000 97.800 88.644 76.248 63.345
46 100.000 98.308 89.395 77.079 64.170
47 100.000 98.308 89.747 77.712 64.922
48 100.000 98.477 90.169 78.312 65.606
49 100.000 98.477 90.286 78.623 66.091
50 100.000 98.646 90.685 79.186 66.738
51 100.000 98.646 90.873 79.583 67.284
52 100.000 98.646 91.084 80.006 67.848
53 100.000 98.646 91.295 80.430 68.412
54 100.000 98.646 91.506 80.848 68.965
55 100.000 98.985 92.046 81.486 69.634
56 100.000 98.985 92.234 81.862 70.139
57 100.000 98.985 92.445 82.280 70.684
58 100.000 98.985 92.586 82.596 71.143
59 100.000 99.154 92.938 83.084 71.709
60 100.000 99.323 93.243 83.513 72.228
61 100.000 99.323 93.430 83.888 72.730
62 100.000 99.323 93.571 84.199 73.175
63 100.000 99.323 93.759 84.569 73.665
64 100.000 99.323 93.876 84.848 74.085
65 100.000 99.323 94.040 85.186 74.547
66 100.000 99.323 94.134 85.432 74.939
67 100.000 99.323 94.252 85.700 75.335
68 100.000 99.323 94.345 85.931 75.694
69 100.000 99.323 94.416 86.129 76.023
70 100.000 99.323 94.557 86.419 76.423
71 100.000 99.323 94.697 86.708 76.822
72 100.000 99.323 94.838 86.998 77.221
73 100.000 99.323 94.955 87.255 77.593
74 100.000 99.323 95.026 87.448 77.909
75 100.000 99.323 95.073 87.609 78.200
76 100.000 99.323 95.143 87.802 78.516
77 100.000 99.492 95.448 88.194 78.958
78 100.000 99.662 95.753 88.585 79.398
79 100.000 99.662 95.847 88.805 79.731
80 100.000 99.831 96.176 89.228 80.193
81 100.000 99.831 96.269 89.443 80.515
82 100.000 99.831 96.340 89.625 80.809
83 100.000 99.831 96.481 89.909 81.188
84 100.000 99.831 96.574 90.124 81.508
85 100.000 99.831 96.715 90.408 81.887
86 100.000 99.831 96.809 90.612 82.187
87 100.000 99.831 96.809 90.698 82.394
88 100.000 99.831 96.903 90.901 82.689
89 100.000 99.831 96.973 91.073 82.958
90 100.000 99.831 97.067 91.271 83.242
91 100.000 99.831 97.137 91.437 83.499
92 100.000 99.831 97.208 91.604 83.756
93 100.000 99.831 97.255 91.738 83.986
94 100.000 99.831 97.302 91.872 84.215
95 100.000 99.831 97.396 92.070 84.496
96 100.000 99.831 97.466 92.231 84.742
97 100.000 99.831 97.536 92.392 84.986
98 100.000 99.831 97.583 92.521 85.204
99 100.000 99.831 97.677 92.714 85.473
100 100.000 99.831 97.771 92.907 85.741
101 100.000 99.831 97.818 93.030 85.948
102 100.000 99.831 97.888 93.185 86.181
103 100.000 99.831 97.935 93.309 86.386
104 100.000 99.831 98.006 93.464 86.617
105 100.000 99.831 98.076 93.620 86.848
106 100.000 99.831 98.146 93.775 87.078
107 100.000 99.831 98.217 93.931 87.307
108 100.000 99.831 98.264 94.049 87.501
109 100.000 99.831 98.264 94.113 87.656
110 100.000 99.831 98.334 94.263 87.873
111 100.000 100.000 98.592 94.558 88.178
112 100.000 100.000 98.639 94.676 88.368
113 100.000 100.000 98.710 94.826 88.585
114 100.000 100.000 98.756 94.939 88.768
115 100.000 100.000 98.780 95.019 88.923
116 100.000 100.000 98.827 95.132 89.103
117 100.000 100.000 98.897 95.276 89.308
118 100.000 100.000 98.921 95.357 89.461
119 100.000 100.000 98.944 95.437 89.615
120 100.000 100.000 99.015 95.582 89.820
121 100.000 100.000 99.061 95.695 89.998
122 100.000 100.000 99.108 95.807 90.175
123 100.000 100.000 99.155 95.920 90.353
124 100.000 100.000 99.202 96.032 90.530
125 100.000 100.000 99.226 96.107 90.673
126 100.000 100.000 99.273 96.215 90.841
127 100.000 100.000 99.273 96.258 90.955
128 100.000 100.000 99.296 96.333 91.094
129 100.000 100.000 99.320 96.408 91.233
130 100.000 100.000 99.366 96.515 91.397
131 100.000 100.000 99.413 96.622 91.562
132 100.000 100.000 99.460 96.729 91.727
133 100.000 100.000 99.460 96.762 91.821
134 100.000 100.000 99.484 96.831 91.947
135 100.000 100.000 99.484 96.869 92.047
136 100.000 100.000 99.507 96.939 92.172
137 100.000 100.000 99.507 96.976 92.272
138 100.000 100.000 99.531 97.046 92.398
139 100.000 100.000 99.554 97.115 92.524
140 100.000 100.000 99.601 97.217 92.673
141 100.000 100.000 99.648 97.319 92.823
142 100.000 100.000 99.672 97.384 92.938
143 100.000 100.000 99.672 97.421 93.034
144 100.000 100.000 99.672 97.453 93.121
145 100.000 100.000 99.672 97.485 93.208
146 100.000 100.000 99.672 97.512 93.287
147 100.000 100.000 99.672 97.544 93.374
148 100.000 100.000 99.672 97.577 93.461
149 100.000 100.000 99.672 97.609 93.548
150 100.000 100.000 99.672 97.641 93.635
151 100.000 100.000 99.695 97.705 93.746
152 100.000 100.000 99.695 97.732 93.822
153 100.000 100.000 99.695 97.759 93.898
154 100.000 100.000 99.695 97.786 93.973
155 100.000 100.000 99.695 97.812 94.049
156 100.000 100.000 99.718 97.871 94.149
157 100.000 100.000 99.718 97.898 94.223
158 100.000 100.000 99.742 97.957 94.323
159 100.000 100.000 99.765 98.016 94.421
160 100.000 100.000 99.765 98.043 94.493
161 100.000 100.000 99.789 98.102 94.591
162 100.000 100.000 99.812 98.161 94.689
163 100.000 100.000 99.836 98.220 94.787
164 100.000 100.000 99.836 98.247 94.859
165 100.000 100.000 99.836 98.274 94.931
166 100.000 100.000 99.836 98.300 95.003
167 100.000 100.000 99.859 98.359 95.101
168 100.000 100.000 99.883 98.418 95.199
169 100.000 100.000 99.906 98.477 95.297
170 100.000 100.000 99.906 98.504 95.370
171 100.000 100.000 99.906 98.526 95.432
172 100.000 100.000 99.906 98.547 95.495
173 100.000 100.000 99.906 98.568 95.558
174 100.000 100.000 99.906 98.590 95.621
175 100.000 100.000 99.906 98.611 95.684
176 100.000 100.000 99.906 98.633 95.745
177 100.000 100.000 99.906 98.654 95.806
178 100.000 100.000 99.906 98.676 95.865
179 100.000 100.000 99.906 98.697 95.924
180 100.000 100.000 99.906 98.719 95.984
181 100.000 100.000 99.906 98.740 96.043
182 100.000 100.000 99.906 98.761 96.102
183 100.000 100.000 99.906 98.783 96.161
184 100.000 100.000 99.906 98.804 96.220
185 100.000 100.000 99.906 98.826 96.279
186 100.000 100.000 99.930 98.879 96.364
187 100.000 100.000 99.930 98.901 96.424
188 100.000 100.000 99.930 98.922 96.483
189 100.000 100.000 99.930 98.944 96.542
190 100.000 100.000 99.930 98.965 96.601
191 100.000 100.000 99.930 98.987 96.660
192 100.000 100.000 99.930 99.008 96.719
193 100.000 100.000 99.953 99.062 96.805
194 100.000 100.000 99.977 99.115 96.888
195 100.000 100.000 100.000 99.169 96.971
196 100.000 100.000 100.000 99.185 97.019
197 100.000 100.000 100.000 99.201 97.067
198 100.000 100.000 100.000 99.217 97.115
199 100.000 100.000 100.000 99.233 97.163
200 100.000 100.000 100.000 99.249 97.211
201 100.000 100.000 100.000 99.265 97.259
202 100.000 100.000 100.000 99.282 97.306
203 100.000 100.000 100.000 99.298 97.352
204 100.000 100.000 100.000 99.314 97.398
205 100.000 100.000 100.000 99.330 97.444
206 100.000 100.000 100.000 99.346 97.491
207 100.000 100.000 100.000 99.362 97.537
208 100.000 100.000 100.000 99.373 97.572
209 100.000 100.000 100.000 99.383 97.607
210 100.000 100.000 100.000 99.394 97.642
211 100.000 100.000 100.000 99.405 97.676
212 100.000 100.000 100.000 99.416 97.709
213 100.000 100.000 100.000 99.426 97.742
214 100.000 100.000 100.000 99.437 97.775
215 100.000 100.000 100.000 99.448 97.809
216 100.000 100.000 100.000 99.458 97.842
217 100.000 100.000 100.000 99.469 97.875
218 100.000 100.000 100.000 99.480 97.909
219 100.000 100.000 100.000 99.491 97.942
220 100.000 100.000 100.000 99.501 97.975
221 100.000 100.000 100.000 99.512 98.007
222 100.000 100.000 100.000 99.523 98.038
223 100.000 100.000 100.000 99.534 98.069
224 100.000 100.000 100.000 99.544 98.101
225 100.000 100.000 100.000 99.555 98.132
226 100.000 100.000 100.000 99.566 98.164
227 100.000 100.000 100.000 99.576 98.195
228 100.000 100.000 100.000 99.587 98.227
229 100.000 100.000 100.000 99.598 98.258
230 100.000 100.000 100.000 99.609 98.289
231 100.000 100.000 100.000 99.619 98.321
232 100.000 100.000 100.000 99.630 98.352
233 100.000 100.000 100.000 99.641 98.384
234 100.000 100.000 100.000 99.651 98.415
235 100.000 100.000 100.000 99.662 98.447
236 100.000 100.000 100.000 99.673 98.478
237 100.000 100.000 100.000 99.684 98.510
238 100.000 100.000 100.000 99.684 98.524
239 100.000 100.000 100.000 99.689 98.547
240 100.000 100.000 100.000 99.694 98.567
241 100.000 100.000 100.000 99.700 98.587
242 100.000 100.000 100.000 99.705 98.608
243 100.000 100.000 100.000 99.710 98.628
244 100.000 100.000 100.000 99.716 98.648
245 100.000 100.000 100.000 99.716 98.659
246 100.000 100.000 100.000 99.721 98.678
247 100.000 100.000 100.000 99.727 98.696
248 100.000 100.000 100.000 99.732 98.715
249 100.000 100.000 100.000 99.737 98.733
250 100.000 100.000 100.000 99.743 98.750
251 100.000 100.000 100.000 99.748 98.767
252 100.000 100.000 100.000 99.753 98.783
253 100.000 100.000 100.000 99.759 98.800
254 100.000 100.000 100.000 99.764 98.817
255 100.000 100.000 100.000 99.769 98.833
256 100.000 100.000 100.000 99.775 98.850
257 100.000 100.000 100.000 99.780 98.866
258 100.000 100.000 100.000 99.786 98.883
259 100.000 100.000 100.000 99.791 98.900
260 100.000 100.000 100.000 99.796 98.916
261 100.000 100.000 100.000 99.802 98.933
262 100.000 100.000 100.000 99.807 98.950
263 100.000 100.000 100.000 99.812 98.966
264 100.000 100.000 100.000 99.818 98.983
265 100.000 100.000 100.000 99.823 99.000
266 100.000 100.000 100.000 99.828 99.016
267 100.000 100.000 100.000 99.834 99.033
268 100.000 100.000 100.000 99.839 99.050
269 100.000 100.000 100.000 99.845 99.066
270 100.000 100.000 100.000 99.850 99.083
271 100.000 100.000 100.000 99.855 99.099
272 100.000 100.000 100.000 99.861 99.116
273 100.000 100.000 100.000 99.866 99.133
274 100.000 100.000 100.000 99.871 99.149
275 100.000 100.000 100.000 99.877 99.166
276 100.000 100.000 100.000 99.882 99.183
277 100.000 100.000 100.000 99.887 99.199
278 100.000 100.000 100.000 99.893 99.216
279 100.000 100.000 100.000 99.898 99.233
280 100.000 100.000 100.000 99.903 99.249
281 100.000 100.000 100.000 99.909 99.266
282 100.000 100.000 100.000 99.914 99.283
283 100.000 100.000 100.000 99.920 99.299
284 100.000 100.000 100.000 99.925 99.316
285 100.000 100.000 100.000 99.930 99.332
286 100.000 100.000 100.000 99.936 99.349
287 100.000 100.000 100.000 99.941 99.366
288 100.000 100.000 100.000 99.946 99.382
289 100.000 100.000 100.000 99.952 99.399
290 100.000 100.000 100.000 99.957 99.416
291 100.000 100.000 100.000 99.962 99.432
292 100.000 100.000 100.000 99.968 99.449
293 100.000 100.000 100.000 99.973 99.466
294 100.000 100.000 100.000 99.979 99.482
295 100.000 100.000 100.000 99.984 99.499
296 100.000 100.000 100.000 99.989 99.516
297 100.000 100.000 100.000 99.995 99.532
298 100.000 100.000 100.000 100.000 99.549
299 100.000 100.000 100.000 100.000 99.556
300 100.000 100.000 100.000 100.000 99.564
301 100.000 100.000 100.000 100.000 99.571
302 100.000 100.000 100.000 100.000 99.578
303 100.000 100.000 100.000 100.000 99.586
304 100.000 100.000 100.000 100.000 99.593
305 100.000 100.000 100.000 100.000 99.601
306 100.000 100.000 100.000 100.000 99.608
307 100.000 100.000 100.000 100.000 99.615
308 100.000 100.000 100.000 100.000 99.623
309 100.000 100.000 100.000 100.000 99.630
310 100.000 100.000 100.000 100.000 99.638
311 100.000 100.000 100.000 100.000 99.643
312 100.000 100.000 100.000 100.000 99.647
313 100.000 100.000 100.000 100.000 99.650
314 100.000 100.000 100.000 100.000 99.654
315 100.000 100.000 100.000 100.000 99.658
316 100.000 100.000 100.000 100.000 99.662
317 100.000 100.000 100.000 100.000 99.665
318 100.000 100.000 100.000 100.000 99.669
319 100.000 100.000 100.000 100.000 99.673
320 100.000 100.000 100.000 100.000 99.676
321 100.000 100.000 100.000 100.000 99.680
322 100.000 100.000 100.000 100.000 99.684
323 100.000 100.000 100.000 100.000 99.687
324 100.000 100.000 100.000 100.000 99.691
325 100.000 100.000 100.000 100.000 99.695
326 100.000 100.000 100.000 100.000 99.699
327 100.000 100.000 100.000 100.000 99.702
328 100.000 100.000 100.000 100.000 99.706
329 100.000 100.000 100.000 100.000 99.710
330 100.000 100.000 100.000 100.000 99.713
331 100.000 100.000 100.000 100.000 99.717
332 100.000 100.000 100.000 100.000 99.721
333 100.000 100.000 100.000 100.000 99.724
334 100.000 100.000 100.000 100.000 99.728
335 100.000 100.000 100.000 100.000 99.732
336 100.000 100.000 100.000 100.000 99.736
337 100.000 100.000 100.000 100.000 99.739
338 100.000 100.000 100.000 100.000 99.743
339 100.000 100.000 100.000 100.000 99.747
340 100.000 100.000 100.000 100.000 99.750
341 100.000 100.000 100.000 100.000 99.754
342 100.000 100.000 100.000 100.000 99.758
343 100.000 100.000 100.000 100.000 99.761
344 100.000 100.000 100.000 100.000 99.765
345 100.000 100.000 100.000 100.000 99.769
346 100.000 100.000 100.000 100.000 99.773
347 100.000 100.000 100.000 100.000 99.776
348 100.000 100.000 100.000 100.000 99.780
349 100.000 100.000 100.000 100.000 99.784
350 100.000 100.000 100.000 100.000 99.787
351 100.000 100.000 100.000 100.000 99.791
352 100.000 100.000 100.000 100.000 99.795
353 100.000 100.000 100.000 100.000 99.798
354 100.000 100.000 100.000 100.000 99.802
355 100.000 100.000 100.000 100.000 99.806
356 100.000 100.000 100.000 100.000 99.810
357 100.000 100.000 100.000 100.000 99.813
358 100.000 100.000 100.000 100.000 99.817
359 100.000 100.000 100.000 100.000 99.821
360 100.000 100.000 100.000 100.000 99.824
361 100.000 100.000 100.000 100.000 99.828
362 100.000 100.000 100.000 100.000 99.832
363 100.000 100.000 100.000 100.000 99.835
364 100.000 100.000 100.000 100.000 99.839
365 100.000 100.000 100.000 100.000 99.843
366 100.000 100.000 100.000 100.000 99.847
367 100.000 100.000 100.000 100.000 99.850
368 100.000 100.000 100.000 100.000 99.854
369 100.000 100.000 100.000 100.000 99.858
370 100.000 100.000 100.000 100.000 99.861
371 100.000 100.000 100.000 100.000 99.865
372 100.000 100.000 100.000 100.000 99.869
373 100.000 100.000 100.000 100.000 99.872
374 100.000 100.000 100.000 100.000 99.876
375 100.000 100.000 100.000 100.000 99.880
376 100.000 100.000 100.000 100.000 99.883
377 100.000 100.000 100.000 100.000 99.887
378 100.000 100.000 100.000 100.000 99.891
379 100.000 100.000 100.000 100.000 99.895
380 100.000 100.000 100.000 100.000 99.896
381 100.000 100.000 100.000 100.000 99.898
382 100.000 100.000 100.000 100.000 99.900
383 100.000 100.000 100.000 100.000 99.902
384 100.000 100.000 100.000 100.000 99.904
385 100.000 100.000 100.000 100.000 99.906
386 100.000 100.000 100.000 100.000 99.908
387 100.000 100.000 100.000 100.000 99.909
388 100.000 100.000 100.000 100.000 99.911
389 100.000 100.000 100.000 100.000 99.913
390 100.000 100.000 100.000 100.000 99.915
391 100.000 100.000 100.000 100.000 99.917
392 100.000 100.000 100.000 100.000 99.919
393 100.000 100.000 100.000 100.000 99.920
394 100.000 100.000 100.000 100.000 99.922
395 100.000 100.000 100.000 100.000 99.924
396 100.000 100.000 100.000 100.000 99.926
397 100.000 100.000 100.000 100.000 99.928
398 100.000 100.000 100.000 100.000 99.930
399 100.000 100.000 100.000 100.000 99.932
400 100.000 100.000 100.000 100.000 99.933
401 100.000 100.000 100.000 100.000 99.935
402 100.000 100.000 100.000 100.000 99.937
403 100.000 100.000 100.000 100.000 99.939
404 100.000 100.000 100.000 100.000 99.941
405 100.000 100.000 100.000 100.000 99.943
406 100.000 100.000 100.000 100.000 99.945
407 100.000 100.000 100.000 100.000 99.946
408 100.000 100.000 100.000 100.000 99.948
409 100.000 100.000 100.000 100.000 99.950
410 100.000 100.000 100.000 100.000 99.952
411 100.000 100.000 100.000 100.000 99.954
412 100.000 100.000 100.000 100.000 99.956
413 100.000 100.000 100.000 100.000 99.957
414 100.000 100.000 100.000 100.000 99.959
415 100.000 100.000 100.000 100.000 99.961
416 100.000 100.000 100.000 100.000 99.963
417 100.000 100.000 100.000 100.000 99.965
418 100.000 100.000 100.000 100.000 99.967
419 100.000 100.000 100.000 100.000 99.969
420 100.000 100.000 100.000 100.000 99.970
421 100.000 100.000 100.000 100.000 99.972
422 100.000 100.000 100.000 100.000 99.974
423 100.000 100.000 100.000 100.000 99.976
424 100.000 100.000 100.000 100.000 99.978
425 100.000 100.000 100.000 100.000 99.980
426 100.000 100.000 100.000 100.000 99.982
427 100.000 100.000 100.000 100.000 99.983
428 100.000 100.000 100.000 100.000 99.985
429 100.000 100.000 100.000 100.000 99.987
430 100.000 100.000 100.000 100.000 99.989
431 100.000 100.000 100.000 100.000 99.991
432 100.000 100.000 100.000 100.000 99.993
433 100.000 100.000 100.000 100.000 99.994
434 100.000 100.000 100.000 100.000 99.996
435 100.000 100.000 100.000 100.000 99.998
436 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.557 67.361 58.835
8 97.368 89.555 80.347 71.159 62.542
9 100.000 93.088 84.092 74.853 66.091
10 100.000 93.856 85.664 77.056 68.721
11 100.000 94.624 87.147 79.050 71.033
12 100.000 95.392 88.616 81.004 73.276
13 100.000 96.160 90.100 82.990 75.565
14 100.000 96.928 91.569 84.927 77.757
15 100.000 97.696 93.038 86.869 79.963
16 100.000 98.464 94.506 88.806 82.154
17 100.000 99.232 96.034 90.906 84.627
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.347 62.970 52.942 44.301
5 89.474 78.648 68.243 58.664 50.095
6 92.105 82.488 72.738 63.484 54.998
7 94.737 86.022 76.601 67.465 58.989
8 97.368 89.555 80.376 71.230 62.647
9 100.000 93.088 84.122 74.918 66.180
10 100.000 93.856 85.679 77.095 68.783
11 100.000 94.624 87.206 79.194 71.258
12 100.000 95.392 88.704 81.219 73.611
13 100.000 96.160 90.203 83.249 75.983
14 100.000 96.928 91.686 85.236 78.272
15 100.000 97.696 93.155 87.177 80.477
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 78.947 61.290 47.474 36.905 28.873
3 84.211 69.585 56.962 46.428 37.793
4 86.842 75.115 64.233 54.358 45.643
5 89.474 78.802 68.478 58.904 50.294
6 92.105 82.488 72.723 63.448 54.940
7 94.737 86.022 76.572 67.396 58.886
8 97.368 89.555 80.405 71.303 62.763
9 100.000 93.088 84.151 74.989 66.290
10 100.000 93.856 85.679 77.093 68.778
11 100.000 94.624 87.206 79.194 71.260
12 100.000 95.392 88.719 81.261 73.688
13 100.000 96.160 90.217 83.294 76.066
14 100.000 96.928 91.686 85.240 78.285
15 100.000 97.696 93.155 87.181 80.490
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 89.474 79.724 70.535 61.935 54.002
7 92.105 83.257 74.354 65.811 57.839
8 94.737 86.790 78.173 69.682 61.660
9 97.368 90.323 81.933 73.411 65.262
10 100.000 93.856 85.693 77.135 68.854
11 100.000 94.624 87.221 79.237 71.335
12 100.000 95.392 88.719 81.272 73.719
13 100.000 96.160 90.217 83.294 76.066
14 100.000 96.928 91.686 85.240 78.285
15 100.000 97.696 93.155 87.181 80.490
16 100.000 98.464 94.624 89.122 82.696
17 100.000 99.232 96.093 91.060 84.886
18 100.000 100.000 97.562 92.997 87.077

gzip

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.347 62.970 52.942 44.301
5 89.474 78.648 68.199 58.561 49.940
6 92.105 82.335 72.429 63.075 54.547
7 94.737 86.022 76.601 67.463 58.983
8 97.368 89.555 80.479 71.477 63.025
9 100.000 93.088 84.269 75.277 66.739
10 100.000 93.702 85.517 77.042 68.885
11 100.000 94.316 86.722 78.683 70.814
12 100.000 94.931 87.985 80.480 73.005
13 100.000 95.545 89.189 82.120 74.929
14 100.000 96.006 90.115 83.420 76.510
15 100.000 96.467 91.040 84.700 78.027
16 100.000 96.928 91.965 85.980 79.541
17 100.000 97.389 92.891 87.260 81.056
18 100.000 97.849 93.816 88.544 82.587
19 100.000 98.157 94.477 89.530 83.849
20 100.000 98.464 95.138 90.500 85.057
21 100.000 98.771 95.799 91.470 86.268
22 100.000 99.078 96.460 92.443 87.489
23 100.000 99.386 97.121 93.419 88.717
24 100.000 99.539 97.488 94.005 89.501
25 100.000 99.693 97.870 94.628 90.342
26 100.000 99.846 98.237 95.212 91.119
27 100.000 100.000 98.605 95.797 91.896
28 100.000 100.000 98.707 96.073 92.373
29 100.000 100.000 98.796 96.310 92.782
30 100.000 100.000 98.884 96.555 93.220
31 100.000 100.000 98.972 96.793 93.630
32 100.000 100.000 99.060 97.034 94.054
33 100.000 100.000 99.163 97.318 94.556
34 100.000 100.000 99.222 97.489 94.871
35 100.000 100.000 99.324 97.764 95.346
36 100.000 100.000 99.412 98.002 95.757
37 100.000 100.000 99.471 98.164 96.046
38 100.000 100.000 99.559 98.401 96.456
39 100.000 100.000 99.618 98.564 96.743
40 100.000 100.000 99.706 98.801 97.153
41 100.000 100.000 99.765 98.963 97.441
42 100.000 100.000 99.824 99.126 97.728
43 100.000 100.000 99.882 99.288 98.015
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.552 71.650 63.284
9 100.000 93.088 84.342 75.451 67.002
10 100.000 93.702 85.590 77.218 69.154
11 100.000 94.316 86.810 78.916 71.203
12 100.000 94.931 88.014 80.568 73.164
13 100.000 95.545 89.219 82.207 75.088
14 100.000 96.160 90.423 83.852 77.030
15 100.000 96.774 91.627 85.483 78.929
16 100.000 97.389 92.832 87.122 80.852
17 100.000 98.003 94.036 88.749 82.737
18 100.000 98.464 94.962 90.037 84.281
19 100.000 98.771 95.623 91.007 85.489
20 100.000 98.925 96.005 91.630 86.331
21 100.000 99.078 96.387 92.267 87.219
22 100.000 99.232 96.769 92.900 88.095
23 100.000 99.386 97.150 93.523 88.938
24 100.000 99.539 97.532 94.147 89.786
25 100.000 99.693 97.914 94.770 90.627
26 100.000 99.846 98.296 95.393 91.468
27 100.000 100.000 98.678 96.016 92.310
28 100.000 100.000 98.766 96.257 92.736
29 100.000 100.000 98.854 96.495 93.146
30 100.000 100.000 98.942 96.732 93.556
31 100.000 100.000 99.031 96.969 93.965
32 100.000 100.000 99.119 97.207 94.377
33 100.000 100.000 99.207 97.444 94.787
34 100.000 100.000 99.295 97.681 95.197
35 100.000 100.000 99.383 97.919 95.607
36 100.000 100.000 99.471 98.156 96.018
37 100.000 100.000 99.530 98.318 96.305
38 100.000 100.000 99.589 98.481 96.593
39 100.000 100.000 99.647 98.643 96.880
40 100.000 100.000 99.706 98.805 97.168
41 100.000 100.000 99.765 98.967 97.455
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.675 67.637 59.246
8 97.368 89.555 80.567 71.684 63.334
9 100.000 93.088 84.327 75.417 66.951
10 100.000 93.702 85.576 77.180 69.091
11 100.000 94.316 86.795 78.878 71.139
12 100.000 94.931 87.999 80.531 73.107
13 100.000 95.545 89.204 82.172 75.036
14 100.000 96.160 90.408 83.813 76.966
15 100.000 96.774 91.613 85.451 78.884
16 100.000 97.389 92.817 87.082 80.782
17 100.000 98.003 94.022 88.709 82.665
18 100.000 98.618 95.226 90.336 84.548
19 100.000 98.771 95.623 91.007 85.488
20 100.000 98.925 96.005 91.634 86.344
21 100.000 99.078 96.387 92.259 87.192
22 100.000 99.232 96.769 92.884 88.041
23 100.000 99.386 97.150 93.506 88.883
24 100.000 99.539 97.532 94.129 89.724
25 100.000 99.693 97.914 94.752 90.565
26 100.000 99.846 98.296 95.375 91.408
27 100.000 100.000 98.678 95.997 92.248
28 100.000 100.000 98.766 96.245 92.692
29 100.000 100.000 98.854 96.486 93.116
30 100.000 100.000 98.942 96.728 93.541
31 100.000 100.000 99.031 96.965 93.951
32 100.000 100.000 99.119 97.202 94.361
33 100.000 100.000 99.207 97.440 94.770
34 100.000 100.000 99.295 97.677 95.180
35 100.000 100.000 99.383 97.915 95.590
36 100.000 100.000 99.471 98.152 96.002
37 100.000 100.000 99.530 98.318 96.304
38 100.000 100.000 99.589 98.481 96.592
39 100.000 100.000 99.647 98.643 96.879
40 100.000 100.000 99.706 98.805 97.167
41 100.000 100.000 99.765 98.967 97.454
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.868 76.293 67.053 58.532
8 97.368 89.401 80.185 71.096 62.610
9 100.000 93.088 84.224 75.173 66.585
10 100.000 93.702 85.532 77.076 68.937
11 100.000 94.316 86.780 78.837 71.071
12 100.000 94.931 87.985 80.493 73.042
13 100.000 95.545 89.189 82.136 74.978
14 100.000 96.160 90.394 83.777 76.909
15 100.000 96.774 91.598 85.410 78.812
16 100.000 97.389 92.803 87.037 80.696
17 100.000 98.003 94.007 88.664 82.580
18 100.000 98.464 94.932 89.950 84.117
19 100.000 98.771 95.593 90.920 85.327
20 100.000 98.925 95.990 91.591 86.267
21 100.000 99.078 96.387 92.263 87.206
22 100.000 99.232 96.769 92.890 88.062
23 100.000 99.386 97.150 93.514 88.911
24 100.000 99.539 97.532 94.137 89.753
25 100.000 99.693 97.914 94.760 90.594
26 100.000 99.846 98.296 95.383 91.434
27 100.000 100.000 98.678 96.006 92.274
28 100.000 100.000 98.766 96.253 92.719
29 100.000 100.000 98.854 96.495 93.144
30 100.000 100.000 98.942 96.732 93.556
31 100.000 100.000 99.031 96.969 93.968
32 100.000 100.000 99.119 97.207 94.378
33 100.000 100.000 99.207 97.444 94.787
34 100.000 100.000 99.295 97.681 95.197
35 100.000 100.000 99.383 97.919 95.607
36 100.000 100.000 99.471 98.156 96.017
37 100.000 100.000 99.530 98.318 96.305
38 100.000 100.000 99.589 98.481 96.593
39 100.000 100.000 99.647 98.643 96.880
40 100.000 100.000 99.706 98.805 97.168
41 100.000 100.000 99.765 98.967 97.455
42 100.000 100.000 99.824 99.130 97.743
43 100.000 100.000 99.882 99.292 98.030
44 100.000 100.000 99.941 99.454 98.318
45 100.000 100.000 100.000 99.617 98.605

gzip

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.150 62.732 54.193
7 94.737 86.022 76.542 67.329 58.789
8 97.368 89.555 80.288 71.013 62.310
9 100.000 93.088 84.122 74.908 66.153
10 100.000 93.856 85.693 77.119 68.805
11 100.000 94.624 87.192 79.152 71.182
12 100.000 95.392 88.690 81.182 73.554
13 100.000 96.160 90.159 83.120 75.744
14 100.000 96.928 91.657 85.142 78.092
15 100.000 97.696 93.126 87.088 80.310
16 100.000 98.464 94.624 89.122 82.693
17 100.000 99.232 96.093 91.068 84.910
18 100.000 100.000 97.562 93.005 87.101
19 100.000 100.000 97.679 93.330 87.673
20 100.000 100.000 97.767 93.581 88.132
21 100.000 100.000 97.855 93.819 88.545
22 100.000 100.000 97.944 94.060 88.969
23 100.000 100.000 98.061 94.383 89.536
24 100.000 100.000 98.135 94.588 89.903
25 100.000 100.000 98.223 94.825 90.316
26 100.000 100.000 98.281 94.987 90.606
27 100.000 100.000 98.370 95.225 91.015
28 100.000 100.000 98.443 95.434 91.397
29 100.000 100.000 98.502 95.596 91.684
30 100.000 100.000 98.575 95.797 92.038
31 100.000 100.000 98.663 96.034 92.449
32 100.000 100.000 98.751 96.271 92.858
33 100.000 100.000 98.840 96.509 93.269
34 100.000 100.000 98.913 96.710 93.621
35 100.000 100.000 98.957 96.835 93.851
36 100.000 100.000 99.031 97.036 94.204
37 100.000 100.000 99.119 97.274 94.616
38 100.000 100.000 99.177 97.436 94.904
39 100.000 100.000 99.236 97.598 95.191
40 100.000 100.000 99.266 97.687 95.363
41 100.000 100.000 99.295 97.777 95.535
42 100.000 100.000 99.310 97.827 95.642
43 100.000 100.000 99.324 97.878 95.750
44 100.000 100.000 99.368 98.004 95.980
45 100.000 100.000 99.383 98.055 96.087
46 100.000 100.000 99.383 98.069 96.137
47 100.000 100.000 99.398 98.119 96.244
48 100.000 100.000 99.486 98.357 96.654
49 100.000 100.000 99.486 98.371 96.703
50 100.000 100.000 99.545 98.533 96.991
51 100.000 100.000 99.574 98.623 97.163
52 100.000 100.000 99.633 98.785 97.450
53 100.000 100.000 99.662 98.874 97.622
54 100.000 100.000 99.706 99.000 97.852
55 100.000 100.000 99.780 99.201 98.204
56 100.000 100.000 99.794 99.251 98.311
57 100.000 100.000 99.838 99.377 98.540
58 100.000 100.000 99.853 99.428 98.648
59 100.000 100.000 99.853 99.442 98.697
60 100.000 100.000 99.882 99.531 98.869
61 100.000 100.000 99.882 99.546 98.919
62 100.000 100.000 99.927 99.671 99.148
63 100.000 100.000 99.956 99.761 99.320
64 100.000 100.000 99.956 99.775 99.370
65 100.000 100.000 99.971 99.826 99.477
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.335 72.459 63.146 54.656
7 94.737 86.022 76.572 67.396 58.886
8 97.368 89.555 80.449 71.408 62.923
9 100.000 93.088 84.254 75.242 66.687
10 100.000 93.856 85.781 77.352 69.194
11 100.000 94.624 87.294 79.423 71.636
12 100.000 95.392 88.793 81.464 74.038
13 100.000 96.160 90.276 83.460 76.359
14 100.000 96.928 91.745 85.422 78.628
15 100.000 97.696 93.214 87.360 80.822
16 100.000 98.464 94.683 89.297 83.015
17 100.000 99.232 96.152 91.234 85.209
18 100.000 100.000 97.620 93.172 87.402
19 100.000 100.000 97.709 93.409 87.817
20 100.000 100.000 97.797 93.665 88.286
21 100.000 100.000 97.885 93.902 88.699
22 100.000 100.000 97.973 94.139 89.112
23 100.000 100.000 98.061 94.381 89.536
24 100.000 100.000 98.149 94.618 89.946
25 100.000 100.000 98.237 94.855 90.357
26 100.000 100.000 98.325 95.097 90.781
27 100.000 100.000 98.414 95.334 91.191
28 100.000 100.000 98.502 95.571 91.601
29 100.000 100.000 98.590 95.809 92.012
30 100.000 100.000 98.678 96.046 92.422
31 100.000 100.000 98.766 96.288 92.846
32 100.000 100.000 98.854 96.525 93.255
33 100.000 100.000 98.942 96.768 93.688
34 100.000 100.000 99.031 97.006 94.098
35 100.000 100.000 99.119 97.243 94.508
36 100.000 100.000 99.207 97.480 94.917
37 100.000 100.000 99.251 97.606 95.147
38 100.000 100.000 99.295 97.732 95.377
39 100.000 100.000 99.339 97.858 95.606
40 100.000 100.000 99.383 97.984 95.836
41 100.000 100.000 99.427 98.109 96.066
42 100.000 100.000 99.471 98.235 96.295
43 100.000 100.000 99.515 98.361 96.525
44 100.000 100.000 99.559 98.487 96.756
45 100.000 100.000 99.603 98.612 96.985
46 100.000 100.000 99.647 98.738 97.216
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.676
49 100.000 100.000 99.780 99.116 97.906
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.365
52 100.000 100.000 99.912 99.493 98.595
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.714 76.058 66.807 58.315
8 97.368 89.247 79.950 70.852 62.398
9 100.000 92.934 83.975 74.896 66.323
10 100.000 93.856 85.752 77.283 69.092
11 100.000 94.624 87.221 79.239 71.341
12 100.000 95.392 88.690 81.194 73.590
13 100.000 96.160 90.159 83.146 75.828
14 100.000 96.928 91.627 85.098 78.066
15 100.000 97.696 93.096 87.039 80.271
16 100.000 98.464 94.565 88.980 82.477
17 100.000 99.232 96.034 90.922 84.681
18 100.000 100.000 97.503 92.863 86.886
19 100.000 100.000 97.650 93.251 87.543
20 100.000 100.000 97.797 93.638 88.200
21 100.000 100.000 97.885 93.888 88.651
22 100.000 100.000 97.973 94.137 89.102
23 100.000 100.000 98.061 94.385 89.547
24 100.000 100.000 98.149 94.632 89.993
25 100.000 100.000 98.237 94.870 90.404
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.228
28 100.000 100.000 98.502 95.582 91.638
29 100.000 100.000 98.590 95.819 92.049
30 100.000 100.000 98.678 96.056 92.460
31 100.000 100.000 98.766 96.294 92.870
32 100.000 100.000 98.854 96.531 93.281
33 100.000 100.000 98.942 96.768 93.692
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.251 97.606 95.151
38 100.000 100.000 99.295 97.732 95.380
39 100.000 100.000 99.339 97.858 95.610
40 100.000 100.000 99.383 97.984 95.840
41 100.000 100.000 99.427 98.109 96.069
42 100.000 100.000 99.471 98.235 96.299
43 100.000 100.000 99.515 98.361 96.529
44 100.000 100.000 99.559 98.487 96.758
45 100.000 100.000 99.603 98.612 96.988
46 100.000 100.000 99.647 98.738 97.218
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.677
49 100.000 100.000 99.780 99.116 97.907
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.366
52 100.000 100.000 99.912 99.493 98.596
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.341 67.685 57.975 49.375
6 92.105 82.181 72.165 62.762 54.233
7 94.737 85.714 76.058 66.807 58.315
8 97.368 89.247 79.950 70.852 62.398
9 100.000 92.934 83.975 74.896 66.323
10 100.000 93.856 85.752 77.283 69.092
11 100.000 94.624 87.221 79.239 71.341
12 100.000 95.392 88.690 81.194 73.590
13 100.000 96.160 90.159 83.146 75.828
14 100.000 96.928 91.627 85.098 78.066
15 100.000 97.696 93.096 87.039 80.271
16 100.000 98.464 94.565 88.980 82.477
17 100.000 99.232 96.034 90.922 84.681
18 100.000 100.000 97.503 92.863 86.886
19 100.000 100.000 97.650 93.251 87.543
20 100.000 100.000 97.797 93.638 88.200
21 100.000 100.000 97.885 93.888 88.651
22 100.000 100.000 97.973 94.137 89.102
23 100.000 100.000 98.061 94.385 89.547
24 100.000 100.000 98.149 94.632 89.993
25 100.000 100.000 98.237 94.870 90.404
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.228
28 100.000 100.000 98.502 95.582 91.639
29 100.000 100.000 98.590 95.819 92.050
30 100.000 100.000 98.678 96.056 92.461
31 100.000 100.000 98.766 96.294 92.871
32 100.000 100.000 98.854 96.531 93.282
33 100.000 100.000 98.942 96.768 93.692
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.251 97.606 95.151
38 100.000 100.000 99.295 97.732 95.380
39 100.000 100.000 99.339 97.858 95.610
40 100.000 100.000 99.383 97.984 95.840
41 100.000 100.000 99.427 98.109 96.069
42 100.000 100.000 99.471 98.235 96.299
43 100.000 100.000 99.515 98.361 96.529
44 100.000 100.000 99.559 98.487 96.758
45 100.000 100.000 99.603 98.612 96.988
46 100.000 100.000 99.647 98.738 97.218
47 100.000 100.000 99.692 98.864 97.447
48 100.000 100.000 99.736 98.990 97.677
49 100.000 100.000 99.780 99.116 97.907
50 100.000 100.000 99.824 99.241 98.136
51 100.000 100.000 99.868 99.367 98.366
52 100.000 100.000 99.912 99.493 98.596
53 100.000 100.000 99.956 99.619 98.825
54 100.000 100.000 100.000 99.744 99.055
55 100.000 100.000 100.000 99.759 99.104
56 100.000 100.000 100.000 99.773 99.154
57 100.000 100.000 100.000 99.787 99.203
58 100.000 100.000 100.000 99.801 99.253
59 100.000 100.000 100.000 99.815 99.302
60 100.000 100.000 100.000 99.830 99.352
61 100.000 100.000 100.000 99.844 99.401
62 100.000 100.000 100.000 99.858 99.451
63 100.000 100.000 100.000 99.872 99.500
64 100.000 100.000 100.000 99.886 99.550
65 100.000 100.000 100.000 99.901 99.599
66 100.000 100.000 100.000 99.915 99.649
67 100.000 100.000 100.000 99.929 99.698
68 100.000 100.000 100.000 99.943 99.748
69 100.000 100.000 100.000 99.957 99.797
70 100.000 100.000 100.000 99.972 99.847
71 100.000 100.000 100.000 99.986 99.896
72 100.000 100.000 100.000 100.000 99.946

gzip

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.449 71.408 62.925
9 100.000 93.088 84.195 75.088 66.434
10 100.000 93.856 85.664 77.030 68.638
11 100.000 94.624 87.147 79.036 70.987
12 100.000 95.392 88.646 81.057 73.327
13 100.000 96.160 90.129 83.051 75.641
14 100.000 96.928 91.627 85.071 77.980
15 100.000 97.696 93.111 87.061 80.282
16 100.000 98.464 94.683 89.279 82.953
17 100.000 99.232 96.152 91.222 85.166
18 100.000 100.000 97.620 93.159 87.358
19 100.000 100.000 97.679 93.318 87.632
20 100.000 100.000 97.767 93.555 88.045
21 100.000 100.000 97.855 93.792 88.455
22 100.000 100.000 97.914 93.963 88.771
23 100.000 100.000 97.973 94.121 89.044
24 100.000 100.000 98.002 94.212 89.223
25 100.000 100.000 98.090 94.458 89.662
26 100.000 100.000 98.179 94.695 90.072
27 100.000 100.000 98.193 94.738 90.152
28 100.000 100.000 98.281 94.975 90.565
29 100.000 100.000 98.340 95.133 90.838
30 100.000 100.000 98.428 95.375 91.264
31 100.000 100.000 98.472 95.505 91.507
32 100.000 100.000 98.531 95.663 91.781
33 100.000 100.000 98.561 95.746 91.933
34 100.000 100.000 98.649 95.983 92.344
35 100.000 100.000 98.678 96.066 92.496
36 100.000 100.000 98.766 96.304 92.906
37 100.000 100.000 98.781 96.355 93.012
38 100.000 100.000 98.810 96.438 93.163
39 100.000 100.000 98.854 96.559 93.381
40 100.000 100.000 98.913 96.722 93.669
41 100.000 100.000 99.001 96.959 94.079
42 100.000 100.000 99.031 97.042 94.230
43 100.000 100.000 99.045 97.085 94.309
44 100.000 100.000 99.104 97.247 94.597
45 100.000 100.000 99.148 97.369 94.812
46 100.000 100.000 99.163 97.411 94.892
47 100.000 100.000 99.207 97.533 95.108
48 100.000 100.000 99.236 97.616 95.261
49 100.000 100.000 99.266 97.700 95.412
50 100.000 100.000 99.354 97.937 95.822
51 100.000 100.000 99.354 97.945 95.850
52 100.000 100.000 99.354 97.949 95.865
53 100.000 100.000 99.354 97.957 95.894
54 100.000 100.000 99.354 97.959 95.901
55 100.000 100.000 99.412 98.117 96.176
56 100.000 100.000 99.412 98.124 96.197
57 100.000 100.000 99.471 98.286 96.485
58 100.000 100.000 99.530 98.448 96.772
59 100.000 100.000 99.589 98.610 97.059
60 100.000 100.000 99.618 98.698 97.224
61 100.000 100.000 99.633 98.740 97.304
62 100.000 100.000 99.633 98.746 97.325
63 100.000 100.000 99.692 98.911 97.620
64 100.000 100.000 99.692 98.915 97.634
65 100.000 100.000 99.692 98.919 97.649
66 100.000 100.000 99.721 99.002 97.800
67 100.000 100.000 99.750 99.085 97.951
68 100.000 100.000 99.780 99.168 98.102
69 100.000 100.000 99.794 99.211 98.182
70 100.000 100.000 99.794 99.215 98.196
71 100.000 100.000 99.794 99.221 98.218
72 100.000 100.000 99.838 99.343 98.434
73 100.000 100.000 99.838 99.347 98.448
74 100.000 100.000 99.868 99.430 98.599
75 100.000 100.000 99.868 99.434 98.614
76 100.000 100.000 99.868 99.434 98.615
77 100.000 100.000 99.882 99.477 98.694
78 100.000 100.000 99.882 99.487 98.730
79 100.000 100.000 99.882 99.493 98.751
80 100.000 100.000 99.882 99.499 98.773
81 100.000 100.000 99.882 99.507 98.801
82 100.000 100.000 99.912 99.590 98.953
83 100.000 100.000 99.912 99.594 98.967
84 100.000 100.000 99.912 99.594 98.968
85 100.000 100.000 99.912 99.598 98.983
86 100.000 100.000 99.912 99.606 99.011
87 100.000 100.000 99.912 99.613 99.033
88 100.000 100.000 99.912 99.613 99.034
89 100.000 100.000 99.912 99.623 99.069
90 100.000 100.000 99.941 99.706 99.220
91 100.000 100.000 99.956 99.753 99.314
92 100.000 100.000 99.956 99.753 99.314
93 100.000 100.000 99.956 99.757 99.329
94 100.000 100.000 99.956 99.761 99.344
95 100.000 100.000 99.956 99.769 99.373
96 100.000 100.000 99.956 99.775 99.394
97 100.000 100.000 99.985 99.858 99.545
98 100.000 100.000 99.985 99.858 99.546
99 100.000 100.000 99.985 99.858 99.547
100 100.000 100.000 99.985 99.864 99.568
101 100.000 100.000 99.985 99.870 99.590
102 100.000 100.000 100.000 99.913 99.669
103 100.000 100.000 100.000 99.919 99.691
104 100.000 100.000 100.000 99.919 99.691
105 100.000 100.000 100.000 99.919 99.692
106 100.000 100.000 100.000 99.921 99.700
107 100.000 100.000 100.000 99.925 99.714
108 100.000 100.000 100.000 99.927 99.722
109 100.000 100.000 100.000 99.931 99.737
110 100.000 100.000 100.000 99.935 99.751
111 100.000 100.000 100.000 99.943 99.780
112 100.000 100.000 100.000 99.945 99.788
113 100.000 100.000 100.000 99.949 99.802
114 100.000 100.000 100.000 99.951 99.810
115 100.000 100.000 100.000 99.955 99.825
116 100.000 100.000 100.000 99.955 99.825
117 100.000 100.000 100.000 99.959 99.840
118 100.000 100.000 100.000 99.959 99.841
119 100.000 100.000 100.000 99.963 99.856
120 100.000 100.000 100.000 99.968 99.870
121 100.000 100.000 100.000 99.976 99.899
122 100.000 100.000 100.000 99.980 99.914
123 100.000 100.000 100.000 99.984 99.929
124 100.000 100.000 100.000 99.984 99.929
125 100.000 100.000 100.000 99.988 99.944
126 100.000 100.000 100.000 99.988 99.945
127 100.000 100.000 100.000 99.988 99.946
128 100.000 100.000 100.000 99.992 99.960
129 100.000 100.000 100.000 99.992 99.961
130 100.000 100.000 100.000 99.992 99.962
131 100.000 100.000 100.000 99.992 99.963
132 100.000 100.000 100.000 99.994 99.970
133 100.000 100.000 100.000 99.994 99.971
134 100.000 100.000 100.000 99.994 99.972
135 100.000 100.000 100.000 99.998 99.986
136 100.000 100.000 100.000 99.998 99.987
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.559 69.501
11 100.000 94.624 87.338 79.511 71.739
12 100.000 95.392 88.807 81.478 74.025
13 100.000 96.160 90.276 83.428 76.257
14 100.000 96.928 91.745 85.390 78.525
15 100.000 97.696 93.214 87.335 80.745
16 100.000 98.464 94.683 89.283 82.970
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.421 87.851
20 100.000 100.000 97.797 93.663 88.276
21 100.000 100.000 97.885 93.900 88.688
22 100.000 100.000 97.973 94.137 89.098
23 100.000 100.000 98.061 94.379 89.523
24 100.000 100.000 98.149 94.616 89.934
25 100.000 100.000 98.237 94.853 90.346
26 100.000 100.000 98.325 95.091 90.757
27 100.000 100.000 98.414 95.328 91.167
28 100.000 100.000 98.502 95.569 91.592
29 100.000 100.000 98.590 95.807 92.002
30 100.000 100.000 98.678 96.044 92.413
31 100.000 100.000 98.766 96.286 92.838
32 100.000 100.000 98.854 96.523 93.248
33 100.000 100.000 98.942 96.764 93.672
34 100.000 100.000 99.031 97.002 94.082
35 100.000 100.000 99.119 97.239 94.492
36 100.000 100.000 99.207 97.476 94.902
37 100.000 100.000 99.266 97.639 95.190
38 100.000 100.000 99.324 97.797 95.464
39 100.000 100.000 99.383 97.959 95.752
40 100.000 100.000 99.442 98.121 96.039
41 100.000 100.000 99.486 98.247 96.269
42 100.000 100.000 99.530 98.373 96.499
43 100.000 100.000 99.574 98.499 96.728
44 100.000 100.000 99.618 98.625 96.958
45 100.000 100.000 99.662 98.750 97.187
46 100.000 100.000 99.706 98.876 97.417
47 100.000 100.000 99.750 99.002 97.647
48 100.000 100.000 99.794 99.128 97.876
49 100.000 100.000 99.824 99.211 98.028
50 100.000 100.000 99.853 99.298 98.193
51 100.000 100.000 99.882 99.385 98.359
52 100.000 100.000 99.912 99.468 98.510
53 100.000 100.000 99.941 99.556 98.675
54 100.000 100.000 99.971 99.639 98.826
55 100.000 100.000 99.985 99.686 98.920
56 100.000 100.000 100.000 99.732 99.013
57 100.000 100.000 100.000 99.736 99.028
58 100.000 100.000 100.000 99.738 99.036
59 100.000 100.000 100.000 99.740 99.044
60 100.000 100.000 100.000 99.742 99.052
61 100.000 100.000 100.000 99.757 99.101
62 100.000 100.000 100.000 99.767 99.137
63 100.000 100.000 100.000 99.767 99.138
64 100.000 100.000 100.000 99.771 99.152
65 100.000 100.000 100.000 99.773 99.160
66 100.000 100.000 100.000 99.777 99.175
67 100.000 100.000 100.000 99.783 99.197
68 100.000 100.000 100.000 99.797 99.246
69 100.000 100.000 100.000 99.801 99.261
70 100.000 100.000 100.000 99.805 99.276
71 100.000 100.000 100.000 99.805 99.277
72 100.000 100.000 100.000 99.813 99.305
73 100.000 100.000 100.000 99.817 99.320
74 100.000 100.000 100.000 99.821 99.335
75 100.000 100.000 100.000 99.821 99.336
76 100.000 100.000 100.000 99.826 99.350
77 100.000 100.000 100.000 99.832 99.372
78 100.000 100.000 100.000 99.834 99.379
79 100.000 100.000 100.000 99.838 99.394
80 100.000 100.000 100.000 99.840 99.402
81 100.000 100.000 100.000 99.840 99.402
82 100.000 100.000 100.000 99.844 99.417
83 100.000 100.000 100.000 99.844 99.418
84 100.000 100.000 100.000 99.848 99.433
85 100.000 100.000 100.000 99.856 99.461
86 100.000 100.000 100.000 99.860 99.476
87 100.000 100.000 100.000 99.862 99.484
88 100.000 100.000 100.000 99.870 99.512
89 100.000 100.000 100.000 99.878 99.541
90 100.000 100.000 100.000 99.878 99.542
91 100.000 100.000 100.000 99.886 99.571
92 100.000 100.000 100.000 99.890 99.585
93 100.000 100.000 100.000 99.899 99.614
94 100.000 100.000 100.000 99.903 99.629
95 100.000 100.000 100.000 99.907 99.643
96 100.000 100.000 100.000 99.907 99.644
97 100.000 100.000 100.000 99.907 99.645
98 100.000 100.000 100.000 99.913 99.666
99 100.000 100.000 100.000 99.913 99.667
100 100.000 100.000 100.000 99.923 99.703
101 100.000 100.000 100.000 99.929 99.724
102 100.000 100.000 100.000 99.935 99.746
103 100.000 100.000 100.000 99.941 99.767
104 100.000 100.000 100.000 99.947 99.789
105 100.000 100.000 100.000 99.955 99.818
106 100.000 100.000 100.000 99.955 99.818
107 100.000 100.000 100.000 99.963 99.847
108 100.000 100.000 100.000 99.963 99.848
109 100.000 100.000 100.000 99.963 99.849
110 100.000 100.000 100.000 99.966 99.856
111 100.000 100.000 100.000 99.968 99.864
112 100.000 100.000 100.000 99.970 99.871
113 100.000 100.000 100.000 99.970 99.872
114 100.000 100.000 100.000 99.972 99.880
115 100.000 100.000 100.000 99.972 99.880
116 100.000 100.000 100.000 99.976 99.895
117 100.000 100.000 100.000 99.980 99.910
118 100.000 100.000 100.000 99.980 99.910
119 100.000 100.000 100.000 99.980 99.911
120 100.000 100.000 100.000 99.980 99.912
121 100.000 100.000 100.000 99.980 99.913
122 100.000 100.000 100.000 99.988 99.941
123 100.000 100.000 100.000 99.988 99.942
124 100.000 100.000 100.000 99.988 99.943
125 100.000 100.000 100.000 99.992 99.958
126 100.000 100.000 100.000 99.992 99.958
127 100.000 100.000 100.000 99.992 99.959
128 100.000 100.000 100.000 99.992 99.960
129 100.000 100.000 100.000 99.992 99.961
130 100.000 100.000 100.000 99.996 99.975
131 100.000 100.000 100.000 99.996 99.976
132 100.000 100.000 100.000 99.996 99.977
133 100.000 100.000 100.000 99.996 99.978
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.563 69.515
11 100.000 94.624 87.338 79.525 71.783
12 100.000 95.392 88.807 81.487 74.051
13 100.000 96.160 90.276 83.438 76.289
14 100.000 96.928 91.745 85.390 78.526
15 100.000 97.696 93.214 87.341 80.762
16 100.000 98.464 94.683 89.289 82.988
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.425 87.865
20 100.000 100.000 97.797 93.667 88.291
21 100.000 100.000 97.885 93.908 88.717
22 100.000 100.000 97.973 94.149 89.143
23 100.000 100.000 98.061 94.391 89.569
24 100.000 100.000 98.149 94.632 89.995
25 100.000 100.000 98.237 94.870 90.406
26 100.000 100.000 98.325 95.107 90.816
27 100.000 100.000 98.414 95.344 91.226
28 100.000 100.000 98.502 95.582 91.636
29 100.000 100.000 98.590 95.819 92.046
30 100.000 100.000 98.678 96.056 92.456
31 100.000 100.000 98.766 96.294 92.866
32 100.000 100.000 98.854 96.531 93.277
33 100.000 100.000 98.942 96.768 93.687
34 100.000 100.000 99.031 97.006 94.097
35 100.000 100.000 99.119 97.243 94.507
36 100.000 100.000 99.207 97.480 94.917
37 100.000 100.000 99.266 97.643 95.205
38 100.000 100.000 99.324 97.805 95.492
39 100.000 100.000 99.383 97.967 95.780
40 100.000 100.000 99.442 98.130 96.067
41 100.000 100.000 99.501 98.292 96.355
42 100.000 100.000 99.545 98.418 96.584
43 100.000 100.000 99.589 98.543 96.814
44 100.000 100.000 99.633 98.669 97.044
45 100.000 100.000 99.677 98.795 97.273
46 100.000 100.000 99.706 98.882 97.438
47 100.000 100.000 99.736 98.969 97.603
48 100.000 100.000 99.765 99.057 97.768
49 100.000 100.000 99.794 99.144 97.934
50 100.000 100.000 99.824 99.231 98.099
51 100.000 100.000 99.853 99.314 98.250
52 100.000 100.000 99.882 99.397 98.401
53 100.000 100.000 99.912 99.481 98.553
54 100.000 100.000 99.941 99.564 98.704
55 100.000 100.000 99.971 99.647 98.855
56 100.000 100.000 99.985 99.694 98.948
57 100.000 100.000 100.000 99.740 99.041
58 100.000 100.000 100.000 99.755 99.091
59 100.000 100.000 100.000 99.769 99.140
60 100.000 100.000 100.000 99.783 99.190
61 100.000 100.000 100.000 99.797 99.239
62 100.000 100.000 100.000 99.807 99.275
63 100.000 100.000 100.000 99.817 99.310
64 100.000 100.000 100.000 99.828 99.346
65 100.000 100.000 100.000 99.838 99.381
66 100.000 100.000 100.000 99.848 99.417
67 100.000 100.000 100.000 99.858 99.452
68 100.000 100.000 100.000 99.868 99.488
69 100.000 100.000 100.000 99.878 99.523
70 100.000 100.000 100.000 99.888 99.559
71 100.000 100.000 100.000 99.899 99.594
72 100.000 100.000 100.000 99.909 99.630
73 100.000 100.000 100.000 99.919 99.665
74 100.000 100.000 100.000 99.929 99.701
75 100.000 100.000 100.000 99.939 99.736
76 100.000 100.000 100.000 99.949 99.773
77 100.000 100.000 100.000 99.959 99.808
78 100.000 100.000 100.000 99.970 99.844
79 100.000 100.000 100.000 99.980 99.879
80 100.000 100.000 100.000 99.988 99.908
81 100.000 100.000 100.000 99.994 99.929
82 100.000 100.000 100.000 100.000 99.951
83 100.000 100.000 100.000 100.000 99.952
84 100.000 100.000 100.000 100.000 99.952
85 100.000 100.000 100.000 100.000 99.954
86 100.000 100.000 100.000 100.000 99.955
87 100.000 100.000 100.000 100.000 99.955
88 100.000 100.000 100.000 100.000 99.956
89 100.000 100.000 100.000 100.000 99.957
90 100.000 100.000 100.000 100.000 99.958
91 100.000 100.000 100.000 100.000 99.959
92 100.000 100.000 100.000 100.000 99.960
93 100.000 100.000 100.000 100.000 99.961
94 100.000 100.000 100.000 100.000 99.961
95 100.000 100.000 100.000 100.000 99.962
96 100.000 100.000 100.000 100.000 99.963
97 100.000 100.000 100.000 100.000 99.964
98 100.000 100.000 100.000 100.000 99.964
99 100.000 100.000 100.000 100.000 99.965
100 100.000 100.000 100.000 100.000 99.967
101 100.000 100.000 100.000 100.000 99.968
102 100.000 100.000 100.000 100.000 99.968
103 100.000 100.000 100.000 100.000 99.969
104 100.000 100.000 100.000 100.000 99.970
105 100.000 100.000 100.000 100.000 99.971
106 100.000 100.000 100.000 100.000 99.971
107 100.000 100.000 100.000 100.000 99.972
108 100.000 100.000 100.000 100.000 99.973
109 100.000 100.000 100.000 100.000 99.974
110 100.000 100.000 100.000 100.000 99.974
111 100.000 100.000 100.000 100.000 99.975
112 100.000 100.000 100.000 100.000 99.976
113 100.000 100.000 100.000 100.000 99.977
114 100.000 100.000 100.000 100.000 99.977
115 100.000 100.000 100.000 100.000 99.978
116 100.000 100.000 100.000 100.000 99.979
117 100.000 100.000 100.000 100.000 99.980
118 100.000 100.000 100.000 100.000 99.980
119 100.000 100.000 100.000 100.000 99.981
120 100.000 100.000 100.000 100.000 99.982
121 100.000 100.000 100.000 100.000 99.983
122 100.000 100.000 100.000 100.000 99.983
123 100.000 100.000 100.000 100.000 99.984
124 100.000 100.000 100.000 100.000 99.985
125 100.000 100.000 100.000 100.000 99.986
126 100.000 100.000 100.000 100.000 99.986
127 100.000 100.000 100.000 100.000 99.987
128 100.000 100.000 100.000 100.000 99.988
129 100.000 100.000 100.000 100.000 99.989
130 100.000 100.000 100.000 100.000 99.989
131 100.000 100.000 100.000 100.000 99.990
132 100.000 100.000 100.000 100.000 99.991
133 100.000 100.000 100.000 100.000 99.992
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

gzip

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 63.158 42.396 29.730 21.556 16.055
2 81.579 63.902 49.471 38.285 29.772
3 84.211 69.278 56.448 45.841 37.228
4 86.842 74.501 63.205 53.185 44.513
5 89.474 78.495 67.994 58.388 49.838
6 92.105 82.488 72.782 63.590 55.158
7 94.737 86.022 76.733 67.775 59.452
8 97.368 89.555 80.684 71.960 63.746
9 100.000 93.088 84.401 75.589 67.208
10 100.000 93.856 85.870 77.563 69.515
11 100.000 94.624 87.338 79.525 71.783
12 100.000 95.392 88.807 81.487 74.051
13 100.000 96.160 90.276 83.438 76.289
14 100.000 96.928 91.745 85.390 78.526
15 100.000 97.696 93.214 87.339 80.758
16 100.000 98.464 94.683 89.289 82.989
17 100.000 99.232 96.152 91.236 85.214
18 100.000 100.000 97.620 93.184 87.438
19 100.000 100.000 97.709 93.425 87.865
20 100.000 100.000 97.797 93.667 88.291
21 100.000 100.000 97.885 93.908 88.717
22 100.000 100.000 97.973 94.149 89.143
23 100.000 100.000 98.061 94.391 89.570
24 100.000 100.000 98.149 94.632 89.996
25 100.000 100.000 98.237 94.870 90.406
26 100.000 100.000 98.325 95.107 90.817
27 100.000 100.000 98.414 95.344 91.227
28 100.000 100.000 98.502 95.582 91.638
29 100.000 100.000 98.590 95.819 92.049
30 100.000 100.000 98.678 96.056 92.459
31 100.000 100.000 98.766 96.294 92.870
32 100.000 100.000 98.854 96.531 93.280
33 100.000 100.000 98.942 96.768 93.691
34 100.000 100.000 99.031 97.006 94.101
35 100.000 100.000 99.119 97.243 94.511
36 100.000 100.000 99.207 97.480 94.921
37 100.000 100.000 99.266 97.643 95.209
38 100.000 100.000 99.324 97.805 95.496
39 100.000 100.000 99.383 97.967 95.783
40 100.000 100.000 99.442 98.130 96.071
41 100.000 100.000 99.486 98.255 96.301
42 100.000 100.000 99.530 98.381 96.530
43 100.000 100.000 99.559 98.468 96.695
44 100.000 100.000 99.589 98.556 96.860
45 100.000 100.000 99.618 98.643 97.025
46 100.000 100.000 99.647 98.730 97.190
47 100.000 100.000 99.677 98.813 97.342
48 100.000 100.000 99.706 98.896 97.493
49 100.000 100.000 99.736 98.980 97.644
50 100.000 100.000 99.765 99.063 97.795
51 100.000 100.000 99.794 99.146 97.946
52 100.000 100.000 99.824 99.229 98.097
53 100.000 100.000 99.853 99.312 98.248
54 100.000 100.000 99.882 99.395 98.399
55 100.000 100.000 99.912 99.479 98.550
56 100.000 100.000 99.927 99.525 98.644
57 100.000 100.000 99.941 99.572 98.737
58 100.000 100.000 99.956 99.619 98.830
59 100.000 100.000 99.971 99.665 98.923
60 100.000 100.000 99.985 99.712 99.017
61 100.000 100.000 100.000 99.759 99.110
62 100.000 100.000 100.000 99.773 99.160
63 100.000 100.000 100.000 99.787 99.209
64 100.000 100.000 100.000 99.797 99.245
65 100.000 100.000 100.000 99.807 99.280
66 100.000 100.000 100.000 99.817 99.316
67 100.000 100.000 100.000 99.828 99.351
68 100.000 100.000 100.000 99.838 99.387
69 100.000 100.000 100.000 99.848 99.422
70 100.000 100.000 100.000 99.858 99.458
71 100.000 100.000 100.000 99.868 99.493
72 100.000 100.000 100.000 99.878 99.529
73 100.000 100.000 100.000 99.888 99.564
74 100.000 100.000 100.000 99.899 99.600
75 100.000 100.000 100.000 99.909 99.635
76 100.000 100.000 100.000 99.919 99.671
77 100.000 100.000 100.000 99.929 99.706
78 100.000 100.000 100.000 99.939 99.742
79 100.000 100.000 100.000 99.947 99.770
80 100.000 100.000 100.000 99.953 99.792
81 100.000 100.000 100.000 99.959 99.813
82 100.000 100.000 100.000 99.966 99.835
83 100.000 100.000 100.000 99.972 99.856
84 100.000 100.000 100.000 99.978 99.878
85 100.000 100.000 100.000 99.984 99.900
86 100.000 100.000 100.000 99.988 99.914
87 100.000 100.000 100.000 99.992 99.929
88 100.000 100.000 100.000 99.996 99.944
89 100.000 100.000 100.000 100.000 99.958
90 100.000 100.000 100.000 100.000 99.959
91 100.000 100.000 100.000 100.000 99.960
92 100.000 100.000 100.000 100.000 99.961
93 100.000 100.000 100.000 100.000 99.961
94 100.000 100.000 100.000 100.000 99.962
95 100.000 100.000 100.000 100.000 99.963
96 100.000 100.000 100.000 100.000 99.964
97 100.000 100.000 100.000 100.000 99.964
98 100.000 100.000 100.000 100.000 99.965
99 100.000 100.000 100.000 100.000 99.966
100 100.000 100.000 100.000 100.000 99.967
101 100.000 100.000 100.000 100.000 99.968
102 100.000 100.000 100.000 100.000 99.968
103 100.000 100.000 100.000 100.000 99.969
104 100.000 100.000 100.000 100.000 99.970
105 100.000 100.000 100.000 100.000 99.971
106 100.000 100.000 100.000 100.000 99.971
107 100.000 100.000 100.000 100.000 99.972
108 100.000 100.000 100.000 100.000 99.973
109 100.000 100.000 100.000 100.000 99.974
110 100.000 100.000 100.000 100.000 99.974
111 100.000 100.000 100.000 100.000 99.975
112 100.000 100.000 100.000 100.000 99.976
113 100.000 100.000 100.000 100.000 99.977
114 100.000 100.000 100.000 100.000 99.977
115 100.000 100.000 100.000 100.000 99.978
116 100.000 100.000 100.000 100.000 99.979
117 100.000 100.000 100.000 100.000 99.980
118 100.000 100.000 100.000 100.000 99.980
119 100.000 100.000 100.000 100.000 99.981
120 100.000 100.000 100.000 100.000 99.982
121 100.000 100.000 100.000 100.000 99.983
122 100.000 100.000 100.000 100.000 99.983
123 100.000 100.000 100.000 100.000 99.984
124 100.000 100.000 100.000 100.000 99.985
125 100.000 100.000 100.000 100.000 99.986
126 100.000 100.000 100.000 100.000 99.986
127 100.000 100.000 100.000 100.000 99.987
128 100.000 100.000 100.000 100.000 99.988
129 100.000 100.000 100.000 100.000 99.989
130 100.000 100.000 100.000 100.000 99.989
131 100.000 100.000 100.000 100.000 99.990
132 100.000 100.000 100.000 100.000 99.991
133 100.000 100.000 100.000 100.000 99.992
134 100.000 100.000 100.000 100.000 99.992
135 100.000 100.000 100.000 100.000 99.993
136 100.000 100.000 100.000 100.000 99.994
137 100.000 100.000 100.000 100.000 99.995
138 100.000 100.000 100.000 100.000 99.995
139 100.000 100.000 100.000 100.000 99.996
140 100.000 100.000 100.000 100.000 99.997
141 100.000 100.000 100.000 100.000 99.998
142 100.000 100.000 100.000 100.000 99.998
143 100.000 100.000 100.000 100.000 99.999
144 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 2, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 57.500 31.395 18.105 11.144 7.277
3 70.000 42.359 25.843 16.375 10.836
4 75.000 49.668 32.050 20.986 14.150
5 82.500 56.478 37.430 25.080 17.220
6 87.500 61.960 41.982 28.594 19.891
7 90.000 66.279 46.182 32.097 22.655
8 92.500 69.601 49.348 34.818 24.901
9 95.000 72.591 52.142 37.237 26.933
10 97.500 75.415 54.852 39.656 29.000
11 100.000 78.239 57.438 41.889 30.886
12 100.000 79.734 59.756 44.345 33.123
13 100.000 81.063 61.577 46.189 34.801
14 100.000 82.392 63.439 48.112 36.577
15 100.000 83.555 65.053 49.770 38.109
16 100.000 84.551 66.377 51.090 39.308
17 100.000 85.548 67.763 52.520 40.640
18 100.000 86.545 69.150 53.951 41.971
19 100.000 87.375 70.536 55.531 43.514
20 100.000 88.206 71.674 56.689 44.578
21 100.000 89.037 72.812 57.846 45.642
22 100.000 89.701 73.846 59.013 46.803
23 100.000 90.365 74.798 60.022 47.765
24 100.000 91.030 75.874 61.266 49.024
25 100.000 91.528 76.619 62.089 49.840
26 100.000 92.027 77.488 63.152 50.964
27 100.000 92.525 78.233 63.975 51.779
28 100.000 93.023 79.019 64.868 52.673
29 100.000 93.522 79.806 65.761 53.567
30 100.000 94.020 80.633 66.741 54.582
31 100.000 94.352 81.233 67.498 55.408
32 100.000 94.684 81.833 68.242 56.199
33 100.000 95.017 82.351 68.838 56.814
34 100.000 95.349 82.909 69.512 57.528
35 100.000 95.681 83.468 70.194 58.265
36 100.000 96.013 84.026 70.868 58.979
37 100.000 96.346 84.606 71.592 59.770
38 100.000 96.678 85.164 72.266 60.484
39 100.000 96.844 85.599 72.894 61.207
40 100.000 97.010 85.992 73.452 61.853
41 100.000 97.176 86.261 73.792 62.245
42 100.000 97.342 86.530 74.114 62.592
43 100.000 97.508 86.944 74.722 63.316
44 100.000 97.674 87.378 75.350 64.039
45 100.000 97.841 87.772 75.921 64.718
46 100.000 98.007 88.041 76.244 65.065
47 100.000 98.173 88.434 76.802 65.711
48 100.000 98.339 88.744 77.224 66.213
49 100.000 98.505 89.075 77.670 66.726
50 100.000 98.671 89.427 78.158 67.294
51 100.000 98.837 89.758 78.605 67.807
52 100.000 99.003 90.068 79.014 68.276
53 100.000 99.169 90.379 79.424 68.746
54 100.000 99.336 90.648 79.763 69.137
55 100.000 99.502 91.082 80.408 69.905
56 100.000 99.668 91.393 80.817 70.374
57 100.000 99.834 91.724 81.264 70.887
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 80.000 55.316 37.016 24.964 17.203
6 85.000 60.465 41.548 28.677 20.122
7 87.500 64.452 45.396 31.915 22.716
8 92.500 69.269 49.328 35.016 25.149
9 95.000 72.924 52.866 38.035 27.599
10 97.500 75.914 55.659 40.454 29.631
11 100.000 78.738 58.225 42.646 31.462
12 100.000 80.066 60.087 44.568 33.238
13 100.000 81.395 61.949 46.491 35.015
14 100.000 82.392 63.542 48.274 36.743
15 100.000 83.389 65.032 49.899 38.315
16 100.000 84.385 66.418 51.329 39.646
17 100.000 85.382 67.805 52.760 40.978
18 100.000 86.213 69.088 54.199 42.404
19 100.000 87.209 70.432 55.560 43.657
20 100.000 88.040 71.612 56.804 44.843
21 100.000 88.870 72.750 57.962 45.907
22 100.000 89.701 73.888 59.120 46.971
23 100.000 90.365 74.922 60.286 48.132
24 100.000 91.030 75.936 61.423 49.270
25 100.000 91.694 76.929 62.519 50.353
26 100.000 92.359 77.881 63.528 51.315
27 100.000 92.857 78.709 64.508 52.330
28 100.000 93.355 79.495 65.401 53.224
29 100.000 93.854 80.240 66.224 54.040
30 100.000 94.020 80.799 67.080 55.043
31 100.000 94.352 81.419 67.874 55.912
32 100.000 94.684 82.019 68.618 56.704
33 100.000 95.017 82.599 69.338 57.484
34 100.000 95.349 83.178 70.045 58.232
35 100.000 95.681 83.737 70.719 58.946
36 100.000 95.847 84.171 71.348 59.669
37 100.000 96.013 84.606 71.976 60.393
38 100.000 96.346 85.123 72.572 61.008
39 100.000 96.678 85.640 73.167 61.623
40 100.000 97.010 86.158 73.763 62.238
41 100.000 97.176 86.571 74.366 62.951
42 100.000 97.342 86.965 74.925 63.597
43 100.000 97.508 87.337 75.462 64.242
44 100.000 97.674 87.709 76.004 64.898
45 100.000 97.841 88.061 76.492 65.465
46 100.000 98.007 88.413 76.971 66.013
47 100.000 98.173 88.765 77.459 66.580
48 100.000 98.339 89.096 77.906 67.093
49 100.000 98.505 89.427 78.353 67.607
50 100.000 98.671 89.737 78.762 68.076
51 100.000 98.837 90.048 79.171 68.545
52 100.000 99.003 90.358 79.585 69.025
53 100.000 99.169 90.668 79.994 69.495
54 100.000 99.336 90.958 80.371 69.933
55 100.000 99.502 91.227 80.693 70.280
56 100.000 99.668 91.496 81.016 70.628
57 100.000 99.834 91.765 81.338 70.976
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 77.500 54.319 36.706 24.881 17.185
6 82.500 59.468 41.237 28.594 20.103
7 85.000 63.289 44.920 31.700 22.596
8 90.000 67.774 48.624 34.723 25.040
9 95.000 72.591 52.555 37.824 27.473
10 97.500 75.415 55.204 40.165 29.480
11 100.000 78.073 57.645 42.319 31.342
12 100.000 79.070 59.197 44.039 33.007
13 100.000 80.233 60.852 45.776 34.638
14 100.000 81.561 62.632 47.550 36.238
15 100.000 82.724 64.246 49.208 37.770
16 100.000 83.721 65.673 50.717 39.200
17 100.000 84.718 67.080 52.190 40.586
18 100.000 85.714 68.487 53.662 41.972
19 100.000 86.545 69.708 54.985 43.256
20 100.000 87.209 70.764 56.192 44.472
21 100.000 88.206 72.088 57.511 45.671
22 100.000 89.037 73.246 58.715 46.802
23 100.000 89.867 74.384 59.873 47.866
24 100.000 90.698 75.522 61.030 48.929
25 100.000 91.196 76.350 62.019 49.965
26 100.000 91.528 77.033 62.924 50.967
27 100.000 92.027 77.840 63.871 51.948
28 100.000 92.691 78.792 64.880 52.910
29 100.000 93.189 79.578 65.790 53.848
30 100.000 93.522 80.219 66.621 54.761
31 100.000 94.020 81.006 67.514 55.654
32 100.000 94.352 81.626 68.304 56.513
33 100.000 94.850 82.371 69.127 57.329
34 100.000 95.183 82.930 69.818 58.086
35 100.000 95.349 83.364 70.446 58.810
36 100.000 95.515 83.799 71.075 59.533
37 100.000 95.681 84.192 71.645 60.212
38 100.000 95.847 84.585 72.204 60.857
39 100.000 96.013 84.978 72.762 61.503
40 100.000 96.179 85.371 73.320 62.148
41 100.000 96.512 85.889 73.916 62.764
42 100.000 96.844 86.406 74.511 63.379
43 100.000 97.176 86.923 75.106 63.994
44 100.000 97.508 87.441 75.702 64.609
45 100.000 97.841 87.958 76.297 65.225
46 100.000 98.007 88.310 76.789 65.805
47 100.000 98.173 88.661 77.277 66.372
48 100.000 98.339 89.013 77.765 66.940
49 100.000 98.505 89.365 78.245 67.487
50 100.000 98.671 89.696 78.692 68.000
51 100.000 98.837 90.027 79.138 68.514
52 100.000 99.003 90.337 79.552 68.994
53 100.000 99.169 90.648 79.961 69.463
54 100.000 99.336 90.958 80.371 69.933
55 100.000 99.502 91.227 80.710 70.324
56 100.000 99.668 91.496 81.032 70.672
57 100.000 99.834 91.765 81.355 71.020
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 2, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 70.000 41.694 25.347 16.123 10.733
4 75.000 48.505 31.181 20.543 13.968
5 75.000 52.658 35.837 24.472 17.008
6 77.500 56.312 39.623 27.838 19.773
7 85.000 62.791 44.527 31.506 22.539
8 90.000 67.608 48.500 34.677 25.050
9 92.500 70.764 51.500 37.282 27.227
10 95.000 73.588 54.149 39.623 29.235
11 95.000 75.083 56.218 41.732 31.157
12 97.500 77.741 58.680 43.923 33.062
13 97.500 79.070 60.501 45.776 34.761
14 100.000 81.728 62.880 47.823 36.501
15 100.000 82.724 64.349 49.411 38.029
16 100.000 83.721 65.798 50.961 39.514
17 100.000 84.551 67.101 52.417 40.935
18 100.000 85.548 68.529 53.926 42.365
19 100.000 86.545 69.915 55.357 43.696
20 100.000 87.209 70.929 56.494 44.834
21 100.000 88.040 72.108 57.739 46.020
22 100.000 88.704 73.102 58.835 47.103
23 100.000 89.037 73.743 59.666 48.016
24 100.000 89.535 74.529 60.576 48.954
25 100.000 90.365 75.667 61.733 50.018
26 100.000 91.196 76.805 62.891 51.082
27 100.000 91.694 77.571 63.760 51.964
28 100.000 92.193 78.336 64.628 52.847
29 100.000 92.857 79.288 65.637 53.809
30 100.000 93.355 80.074 66.530 54.702
31 100.000 93.854 80.861 67.423 55.596
32 100.000 94.186 81.440 68.143 56.377
33 100.000 94.518 82.019 68.862 57.157
34 100.000 94.850 82.619 69.607 57.949
35 100.000 95.017 83.054 70.235 58.672
36 100.000 95.183 83.426 70.773 59.318
37 100.000 95.515 84.006 71.480 60.066
38 100.000 95.847 84.564 72.154 60.779
39 100.000 96.013 84.958 72.712 61.425
40 100.000 96.179 85.351 73.270 62.071
41 100.000 96.346 85.744 73.829 62.716
42 100.000 96.512 86.096 74.317 63.284
43 100.000 96.844 86.613 74.912 63.899
44 100.000 97.176 87.130 75.508 64.515
45 100.000 97.508 87.647 76.103 65.130
46 100.000 97.841 88.165 76.699 65.745
47 100.000 98.173 88.682 77.294 66.360
48 100.000 98.339 89.013 77.749 66.894
49 100.000 98.505 89.365 78.229 67.441
50 100.000 98.671 89.696 78.675 67.954
51 100.000 98.837 90.006 79.089 68.435
52 100.000 99.003 90.317 79.498 68.904
53 100.000 99.169 90.627 79.907 69.373
54 100.000 99.336 90.937 80.317 69.843
55 100.000 99.502 91.227 80.693 70.280
56 100.000 99.668 91.496 81.032 70.672
57 100.000 99.834 91.765 81.355 71.020
58 100.000 100.000 92.034 81.677 71.367

sed

Generation Strength: 3, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 85.000 56.146 36.478 24.286 16.699
6 85.000 60.299 41.134 28.214 19.739
7 90.000 65.449 45.665 31.927 22.657
8 92.500 69.435 49.514 35.165 25.251
9 95.000 72.757 52.700 37.919 27.531
10 97.500 75.748 55.493 40.338 29.563
11 100.000 78.405 57.914 42.460 31.390
12 100.000 79.734 59.735 44.304 33.068
13 100.000 81.063 61.556 46.148 34.746
14 100.000 82.392 63.418 48.071 36.523
15 100.000 83.555 65.156 49.915 38.236
16 100.000 84.718 66.770 51.573 39.768
17 100.000 85.714 68.156 53.004 41.099
18 100.000 86.711 69.584 54.514 42.529
19 100.000 87.542 70.722 55.671 43.593
20 100.000 88.372 72.026 57.143 45.053
21 100.000 89.203 73.164 58.301 46.117
22 100.000 90.033 74.302 59.459 47.181
23 100.000 90.698 75.336 60.634 48.363
24 100.000 91.362 76.371 61.800 49.524
25 100.000 92.027 77.364 62.895 50.607
26 100.000 92.525 78.274 64.024 51.799
27 100.000 93.023 79.040 64.893 52.682
28 100.000 93.522 79.826 65.802 53.619
29 100.000 94.020 80.612 66.696 54.513
30 100.000 94.352 81.212 67.448 55.327
31 100.000 94.684 81.730 68.044 55.943
32 100.000 95.017 82.288 68.718 56.656
33 100.000 95.349 82.888 69.462 57.448
34 100.000 95.681 83.489 70.215 58.263
35 100.000 96.013 84.047 70.897 58.999
36 100.000 96.346 84.564 71.492 59.615
37 100.000 96.512 84.833 71.823 59.985
38 100.000 96.678 85.206 72.357 60.620
39 100.000 96.844 85.496 72.725 61.035
40 100.000 97.010 85.930 73.370 61.802
41 100.000 97.176 86.323 73.945 62.491
42 100.000 97.342 86.716 74.503 63.137
43 100.000 97.508 87.109 75.061 63.783
44 100.000 97.674 87.399 75.429 64.197
45 100.000 97.841 87.813 76.041 64.930
46 100.000 98.007 88.185 76.566 65.542
47 100.000 98.173 88.496 76.984 66.032
48 100.000 98.339 88.827 77.430 66.545
49 100.000 98.505 89.179 77.918 67.113
50 100.000 98.671 89.448 78.257 67.504
51 100.000 98.837 89.758 78.667 67.974
52 100.000 99.003 90.068 79.076 68.443
53 100.000 99.169 90.420 79.564 69.011
54 100.000 99.336 90.772 80.052 69.579
55 100.000 99.502 91.165 80.623 70.257
56 100.000 99.668 91.517 81.111 70.825
57 100.000 99.834 91.786 81.433 71.173
58 100.000 100.000 92.055 81.764 71.544
59 100.000 100.000 92.158 81.987 71.867
60 100.000 100.000 92.220 82.124 72.069
61 100.000 100.000 92.262 82.219 72.216
62 100.000 100.000 92.882 83.273 73.432
63 100.000 100.000 93.503 84.328 74.648
64 100.000 100.000 93.710 84.745 75.214
65 100.000 100.000 93.834 85.006 75.581
66 100.000 100.000 93.896 85.142 75.783
67 100.000 100.000 94.103 85.535 76.288
68 100.000 100.000 94.269 85.883 76.777
69 100.000 100.000 94.413 86.185 77.199
70 100.000 100.000 94.455 86.280 77.346
71 100.000 100.000 94.682 86.722 77.923
72 100.000 100.000 94.765 86.904 78.192
73 100.000 100.000 94.807 86.999 78.339
74 100.000 100.000 94.972 87.346 78.828
75 100.000 100.000 95.034 87.483 79.030
76 100.000 100.000 95.076 87.578 79.177
77 100.000 100.000 95.241 87.909 79.622
78 100.000 100.000 95.303 88.045 79.824
79 100.000 100.000 95.345 88.140 79.971
80 100.000 100.000 95.448 88.368 80.304
81 100.000 100.000 95.593 88.670 80.726
82 100.000 100.000 95.634 88.773 80.896
83 100.000 100.000 95.717 88.947 81.142
84 100.000 100.000 95.820 89.187 81.509
85 100.000 100.000 95.903 89.369 81.778
86 100.000 100.000 95.924 89.422 81.870
87 100.000 100.000 95.986 89.567 82.095
88 100.000 100.000 96.027 89.662 82.242
89 100.000 100.000 96.069 89.757 82.389
90 100.000 100.000 96.131 89.894 82.591
91 100.000 100.000 96.172 89.993 82.750
92 100.000 100.000 96.234 90.134 82.964
93 100.000 100.000 96.338 90.353 83.277
94 100.000 100.000 96.379 90.448 83.424
95 100.000 100.000 96.420 90.543 83.571
96 100.000 100.000 96.462 90.638 83.718
97 100.000 100.000 96.524 90.787 83.955
98 100.000 100.000 96.586 90.923 84.157
99 100.000 100.000 96.648 91.060 84.359
100 100.000 100.000 96.689 91.155 84.506
101 100.000 100.000 96.710 91.205 84.586
102 100.000 100.000 96.814 91.428 84.910
103 100.000 100.000 96.855 91.523 85.057
104 100.000 100.000 96.917 91.668 85.282
105 100.000 100.000 96.938 91.721 85.374
106 100.000 100.000 96.979 91.817 85.521
107 100.000 100.000 97.103 92.077 85.889
108 100.000 100.000 97.145 92.172 86.036
109 100.000 100.000 97.186 92.267 86.183
110 100.000 100.000 97.289 92.491 86.506
111 100.000 100.000 97.352 92.631 86.720
112 100.000 100.000 97.414 92.784 86.966
113 100.000 100.000 97.434 92.838 87.058
114 100.000 100.000 97.496 92.999 87.327
115 100.000 100.000 97.538 93.103 87.497
116 100.000 100.000 97.579 93.198 87.644
117 100.000 100.000 97.600 93.251 87.737
118 100.000 100.000 97.662 93.388 87.938
119 100.000 100.000 97.724 93.524 88.140
120 100.000 100.000 97.765 93.636 88.331
121 100.000 100.000 97.827 93.772 88.533
122 100.000 100.000 97.848 93.822 88.613
123 100.000 100.000 97.910 93.959 88.815
124 100.000 100.000 97.931 94.021 88.931
125 100.000 100.000 98.034 94.240 89.243
126 100.000 100.000 98.076 94.335 89.390
127 100.000 100.000 98.283 94.728 89.895
128 100.000 100.000 98.324 94.823 90.042
129 100.000 100.000 98.345 94.881 90.145
130 100.000 100.000 98.365 94.951 90.281
131 100.000 100.000 98.407 95.046 90.429
132 100.000 100.000 98.448 95.141 90.576
133 100.000 100.000 98.510 95.294 90.821
134 100.000 100.000 98.572 95.431 91.023
135 100.000 100.000 98.614 95.526 91.170
136 100.000 100.000 98.655 95.621 91.317
137 100.000 100.000 98.696 95.716 91.465
138 100.000 100.000 98.738 95.828 91.655
139 100.000 100.000 98.800 95.964 91.857
140 100.000 100.000 98.821 96.018 91.950
141 100.000 100.000 98.862 96.130 92.141
142 100.000 100.000 98.883 96.183 92.233
143 100.000 100.000 98.903 96.237 92.325
144 100.000 100.000 98.965 96.373 92.527
145 100.000 100.000 99.007 96.477 92.697
146 100.000 100.000 99.028 96.526 92.778
147 100.000 100.000 99.069 96.622 92.925
148 100.000 100.000 99.110 96.717 93.072
149 100.000 100.000 99.152 96.828 93.263
150 100.000 100.000 99.234 97.027 93.575
151 100.000 100.000 99.255 97.081 93.668
152 100.000 100.000 99.297 97.176 93.815
153 100.000 100.000 99.317 97.238 93.930
154 100.000 100.000 99.359 97.333 94.077
155 100.000 100.000 99.379 97.387 94.170
156 100.000 100.000 99.421 97.490 94.340
157 100.000 100.000 99.462 97.585 94.487
158 100.000 100.000 99.545 97.767 94.756
159 100.000 100.000 99.565 97.821 94.848
160 100.000 100.000 99.586 97.875 94.941
161 100.000 100.000 99.607 97.924 95.021
162 100.000 100.000 99.628 97.978 95.113
163 100.000 100.000 99.669 98.073 95.261
164 100.000 100.000 99.690 98.135 95.376
165 100.000 100.000 99.710 98.193 95.479
166 100.000 100.000 99.752 98.288 95.626
167 100.000 100.000 99.855 98.511 95.950
168 100.000 100.000 99.897 98.615 96.120
169 100.000 100.000 99.959 98.755 96.334
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 75.000 48.837 31.450 20.692 14.035
5 80.000 55.648 37.285 25.113 17.270
6 87.500 62.458 42.541 29.045 20.207
7 90.000 65.781 45.976 32.147 22.801
8 92.500 69.435 49.555 35.235 25.329
9 95.000 72.425 52.473 37.841 27.542
10 97.500 75.249 55.121 40.181 29.550
11 97.500 76.910 57.397 42.472 31.608
12 100.000 79.568 59.797 44.556 33.392
13 100.000 81.063 61.825 46.595 35.236
14 100.000 82.558 63.811 48.555 36.982
15 100.000 83.555 65.529 50.490 38.825
16 100.000 84.884 67.308 52.264 40.426
17 100.000 86.047 68.881 53.852 41.880
18 100.000 87.043 70.308 55.361 43.310
19 100.000 88.040 71.695 56.800 44.662
20 100.000 89.037 73.081 58.231 45.993
21 100.000 89.867 74.281 59.513 47.222
22 100.000 90.698 75.419 60.671 48.286
23 100.000 91.196 76.329 61.808 49.498
24 100.000 91.860 77.323 62.904 50.582
25 100.000 91.860 77.943 63.958 51.798
26 100.000 92.359 78.771 64.938 52.813
27 100.000 92.857 79.599 65.918 53.828
28 100.000 93.355 80.426 66.898 54.843
29 100.000 93.854 81.233 67.853 55.845
30 100.000 94.186 81.937 68.796 56.891
31 100.000 94.518 82.599 69.661 57.838
32 100.000 94.850 83.220 70.454 58.708
33 100.000 95.183 83.840 71.253 59.587
34 100.000 95.515 84.440 72.009 60.412
35 100.000 95.847 84.999 72.683 61.126
36 100.000 96.013 85.433 73.328 61.893
37 100.000 96.179 85.868 73.957 62.617
38 100.000 96.346 86.303 74.585 63.340
39 100.000 96.512 86.737 75.231 64.107
40 100.000 96.678 87.172 75.871 64.864
41 100.000 96.844 87.565 76.430 65.509
42 100.000 97.010 87.958 76.988 66.155
43 100.000 97.176 88.351 77.546 66.800
44 100.000 97.342 88.703 78.034 67.368
45 100.000 97.508 89.054 78.522 67.936
46 100.000 97.674 89.406 79.010 68.504
47 100.000 97.841 89.758 79.502 69.084
48 100.000 98.007 90.110 79.990 69.652
49 100.000 98.173 90.461 80.478 70.219
50 100.000 98.339 90.813 80.966 70.787
51 100.000 98.505 91.165 81.454 71.355
52 100.000 98.671 91.496 81.901 71.868
53 100.000 98.837 91.827 82.347 72.381
54 100.000 99.003 92.158 82.810 72.938
55 100.000 99.169 92.489 83.265 73.474
56 100.000 99.336 92.820 83.716 73.998
57 100.000 99.502 93.131 84.125 74.468
58 100.000 99.668 93.441 84.543 74.958
59 100.000 99.834 93.751 84.952 75.427
60 100.000 100.000 94.062 85.370 75.917
61 100.000 100.000 94.269 85.763 76.422
62 100.000 100.000 94.475 86.156 76.926
63 100.000 100.000 94.682 86.548 77.431
64 100.000 100.000 94.827 86.859 77.876
65 100.000 100.000 94.972 87.169 78.321
66 100.000 100.000 95.096 87.429 78.688
67 100.000 100.000 95.220 87.690 79.055
68 100.000 100.000 95.345 87.950 79.422
69 100.000 100.000 95.448 88.178 79.758
70 100.000 100.000 95.551 88.401 80.081
71 100.000 100.000 95.655 88.624 80.405
72 100.000 100.000 95.758 88.848 80.728
73 100.000 100.000 95.841 89.042 81.030
74 100.000 100.000 95.924 89.224 81.299
75 100.000 100.000 96.007 89.406 81.567
76 100.000 100.000 96.089 89.604 81.880
77 100.000 100.000 96.172 89.786 82.148
78 100.000 100.000 96.255 89.968 82.417
79 100.000 100.000 96.317 90.105 82.619
80 100.000 100.000 96.379 90.245 82.833
81 100.000 100.000 96.441 90.382 83.035
82 100.000 100.000 96.503 90.522 83.249
83 100.000 100.000 96.565 90.667 83.474
84 100.000 100.000 96.627 90.820 83.719
85 100.000 100.000 96.689 90.965 83.944
86 100.000 100.000 96.752 91.114 84.179
87 100.000 100.000 96.814 91.267 84.425
88 100.000 100.000 96.876 91.403 84.626
89 100.000 100.000 96.938 91.540 84.828
90 100.000 100.000 97.000 91.676 85.030
91 100.000 100.000 97.062 91.812 85.232
92 100.000 100.000 97.124 91.949 85.434
93 100.000 100.000 97.186 92.085 85.636
94 100.000 100.000 97.248 92.222 85.837
95 100.000 100.000 97.310 92.358 86.039
96 100.000 100.000 97.372 92.495 86.241
97 100.000 100.000 97.434 92.631 86.443
98 100.000 100.000 97.496 92.768 86.645
99 100.000 100.000 97.558 92.904 86.847
100 100.000 100.000 97.621 93.041 87.048
101 100.000 100.000 97.683 93.177 87.250
102 100.000 100.000 97.745 93.313 87.452
103 100.000 100.000 97.807 93.466 87.698
104 100.000 100.000 97.869 93.603 87.900
105 100.000 100.000 97.931 93.739 88.101
106 100.000 100.000 97.993 93.876 88.303
107 100.000 100.000 98.055 94.012 88.505
108 100.000 100.000 98.117 94.149 88.707
109 100.000 100.000 98.158 94.244 88.854
110 100.000 100.000 98.200 94.347 89.024
111 100.000 100.000 98.241 94.447 89.184
112 100.000 100.000 98.283 94.542 89.331
113 100.000 100.000 98.324 94.653 89.522
114 100.000 100.000 98.365 94.765 89.712
115 100.000 100.000 98.407 94.868 89.883
116 100.000 100.000 98.448 94.963 90.030
117 100.000 100.000 98.490 95.075 90.221
118 100.000 100.000 98.531 95.170 90.368
119 100.000 100.000 98.572 95.265 90.515
120 100.000 100.000 98.614 95.360 90.662
121 100.000 100.000 98.655 95.455 90.809
122 100.000 100.000 98.696 95.559 90.979
123 100.000 100.000 98.738 95.654 91.127
124 100.000 100.000 98.779 95.749 91.274
125 100.000 100.000 98.821 95.844 91.421
126 100.000 100.000 98.862 95.943 91.580
127 100.000 100.000 98.903 96.047 91.750
128 100.000 100.000 98.945 96.142 91.897
129 100.000 100.000 98.986 96.241 92.057
130 100.000 100.000 99.028 96.336 92.204
131 100.000 100.000 99.069 96.431 92.351
132 100.000 100.000 99.110 96.526 92.498
133 100.000 100.000 99.152 96.622 92.645
134 100.000 100.000 99.193 96.717 92.792
135 100.000 100.000 99.234 96.812 92.939
136 100.000 100.000 99.276 96.907 93.087
137 100.000 100.000 99.317 97.019 93.277
138 100.000 100.000 99.338 97.089 93.414
139 100.000 100.000 99.359 97.151 93.529
140 100.000 100.000 99.379 97.213 93.645
141 100.000 100.000 99.400 97.275 93.760
142 100.000 100.000 99.421 97.337 93.876
143 100.000 100.000 99.441 97.391 93.968
144 100.000 100.000 99.462 97.440 94.048
145 100.000 100.000 99.483 97.498 94.153
146 100.000 100.000 99.503 97.556 94.256
147 100.000 100.000 99.524 97.614 94.360
148 100.000 100.000 99.545 97.664 94.440
149 100.000 100.000 99.565 97.726 94.555
150 100.000 100.000 99.586 97.775 94.636
151 100.000 100.000 99.607 97.829 94.728
152 100.000 100.000 99.628 97.879 94.808
153 100.000 100.000 99.648 97.932 94.901
154 100.000 100.000 99.669 97.982 94.981
155 100.000 100.000 99.690 98.036 95.073
156 100.000 100.000 99.710 98.090 95.166
157 100.000 100.000 99.731 98.139 95.246
158 100.000 100.000 99.752 98.193 95.338
159 100.000 100.000 99.772 98.247 95.431
160 100.000 100.000 99.793 98.300 95.523
161 100.000 100.000 99.814 98.362 95.639
162 100.000 100.000 99.834 98.416 95.731
163 100.000 100.000 99.855 98.470 95.823
164 100.000 100.000 99.876 98.524 95.916
165 100.000 100.000 99.897 98.573 95.996
166 100.000 100.000 99.917 98.627 96.089
167 100.000 100.000 99.938 98.681 96.181
168 100.000 100.000 99.959 98.743 96.296
169 100.000 100.000 99.979 98.797 96.389
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 43.023 25.926 16.354 10.818
4 80.000 50.166 31.533 20.552 13.924
5 80.000 54.319 36.189 24.480 16.964
6 85.000 60.133 41.196 28.417 19.960
7 87.500 64.452 45.438 31.973 22.769
8 92.500 69.435 49.576 35.260 25.347
9 95.000 72.591 52.576 37.865 27.525
10 97.500 75.581 55.411 40.355 29.635
11 100.000 78.405 58.059 42.695 31.642
12 100.000 79.568 59.921 44.717 33.528
13 100.000 80.731 61.701 46.653 35.365
14 100.000 82.226 63.687 48.613 37.111
15 100.000 83.056 65.115 50.271 38.753
16 100.000 84.385 66.915 52.082 40.396
17 100.000 85.548 68.487 53.670 41.851
18 100.000 86.711 70.039 55.241 43.305
19 100.000 87.874 71.612 56.829 44.759
20 100.000 87.874 72.233 57.884 45.975
21 100.000 88.704 73.453 59.207 47.259
22 100.000 89.535 74.674 60.530 48.543
23 100.000 90.199 75.729 61.742 49.768
24 100.000 91.030 76.888 62.929 50.858
25 100.000 91.528 77.736 63.946 51.917
26 100.000 92.027 78.585 64.963 52.976
27 100.000 92.359 79.247 65.844 53.962
28 100.000 92.857 80.033 66.754 54.899
29 100.000 93.355 80.819 67.663 55.837
30 100.000 93.688 81.440 68.465 56.727
31 100.000 94.020 82.061 69.259 57.596
32 100.000 94.352 82.682 70.053 58.466
33 100.000 94.684 83.282 70.806 59.280
34 100.000 94.850 83.716 71.451 60.047
35 100.000 95.017 84.151 72.096 60.815
36 100.000 95.183 84.585 72.741 61.582
37 100.000 95.515 85.144 73.415 62.296
38 100.000 95.847 85.702 74.089 63.009
39 100.000 96.013 86.116 74.685 63.699
40 100.000 96.179 86.509 75.251 64.367
41 100.000 96.512 87.047 75.888 65.037
42 100.000 96.844 87.585 76.525 65.707
43 100.000 97.176 88.123 77.162 66.377
44 100.000 97.342 88.516 77.720 67.023
45 100.000 97.508 88.910 78.278 67.669
46 100.000 97.674 89.303 78.836 68.314
47 100.000 97.841 89.696 79.395 68.960
48 100.000 98.007 90.048 79.891 69.551
49 100.000 98.173 90.399 80.379 70.119
50 100.000 98.339 90.751 80.867 70.686
51 100.000 98.505 91.082 81.322 71.220
52 100.000 98.671 91.413 81.776 71.754
53 100.000 98.837 91.744 82.231 72.290
54 100.000 99.003 92.075 82.686 72.826
55 100.000 99.169 92.406 83.141 73.363
56 100.000 99.336 92.737 83.596 73.896
57 100.000 99.336 92.944 83.989 74.401
58 100.000 99.336 93.151 84.382 74.905
59 100.000 99.336 93.358 84.774 75.410
60 100.000 99.502 93.689 85.221 75.923
61 100.000 99.668 94.020 85.668 76.436
62 100.000 99.834 94.331 86.089 76.938
63 100.000 100.000 94.641 86.499 77.408
64 100.000 100.000 94.765 86.780 77.830
65 100.000 100.000 94.889 87.057 78.241
66 100.000 100.000 95.013 87.326 78.631
67 100.000 100.000 95.138 87.586 78.998
68 100.000 100.000 95.241 87.814 79.334
69 100.000 100.000 95.345 88.037 79.657
70 100.000 100.000 95.448 88.260 79.981
71 100.000 100.000 95.551 88.484 80.304
72 100.000 100.000 95.655 88.707 80.627
73 100.000 100.000 95.758 88.926 80.940
74 100.000 100.000 95.841 89.112 81.221
75 100.000 100.000 95.924 89.298 81.502
76 100.000 100.000 96.007 89.480 81.770
77 100.000 100.000 96.089 89.662 82.039
78 100.000 100.000 96.172 89.844 82.308
79 100.000 100.000 96.255 90.026 82.576
80 100.000 100.000 96.338 90.208 82.845
81 100.000 100.000 96.420 90.390 83.114
82 100.000 100.000 96.483 90.543 83.359
83 100.000 100.000 96.545 90.696 83.605
84 100.000 100.000 96.607 90.849 83.851
85 100.000 100.000 96.669 90.998 84.085
86 100.000 100.000 96.731 91.143 84.310
87 100.000 100.000 96.793 91.287 84.535
88 100.000 100.000 96.855 91.432 84.760
89 100.000 100.000 96.917 91.577 84.985
90 100.000 100.000 96.979 91.717 85.199
91 100.000 100.000 97.041 91.858 85.413
92 100.000 100.000 97.103 91.994 85.615
93 100.000 100.000 97.165 92.131 85.817
94 100.000 100.000 97.227 92.267 86.019
95 100.000 100.000 97.289 92.404 86.220
96 100.000 100.000 97.352 92.540 86.422
97 100.000 100.000 97.414 92.677 86.624
98 100.000 100.000 97.476 92.813 86.826
99 100.000 100.000 97.538 92.950 87.028
100 100.000 100.000 97.600 93.086 87.230
101 100.000 100.000 97.662 93.223 87.431
102 100.000 100.000 97.724 93.359 87.633
103 100.000 100.000 97.786 93.495 87.835
104 100.000 100.000 97.848 93.632 88.037
105 100.000 100.000 97.910 93.768 88.239
106 100.000 100.000 97.972 93.905 88.441
107 100.000 100.000 98.034 94.041 88.642
108 100.000 100.000 98.076 94.153 88.833
109 100.000 100.000 98.117 94.265 89.024
110 100.000 100.000 98.158 94.368 89.194
111 100.000 100.000 98.200 94.471 89.365
112 100.000 100.000 98.241 94.575 89.535
113 100.000 100.000 98.283 94.678 89.705
114 100.000 100.000 98.324 94.781 89.875
115 100.000 100.000 98.365 94.877 90.022
116 100.000 100.000 98.407 94.972 90.170
117 100.000 100.000 98.448 95.067 90.317
118 100.000 100.000 98.490 95.162 90.464
119 100.000 100.000 98.531 95.257 90.611
120 100.000 100.000 98.572 95.352 90.758
121 100.000 100.000 98.614 95.447 90.905
122 100.000 100.000 98.655 95.542 91.052
123 100.000 100.000 98.696 95.637 91.199
124 100.000 100.000 98.738 95.733 91.347
125 100.000 100.000 98.779 95.828 91.494
126 100.000 100.000 98.821 95.923 91.641
127 100.000 100.000 98.862 96.018 91.788
128 100.000 100.000 98.903 96.113 91.935
129 100.000 100.000 98.945 96.208 92.082
130 100.000 100.000 98.986 96.303 92.229
131 100.000 100.000 99.028 96.398 92.376
132 100.000 100.000 99.069 96.493 92.524
133 100.000 100.000 99.110 96.589 92.671
134 100.000 100.000 99.152 96.684 92.818
135 100.000 100.000 99.193 96.779 92.965
136 100.000 100.000 99.234 96.874 93.112
137 100.000 100.000 99.276 96.969 93.259
138 100.000 100.000 99.317 97.064 93.406
139 100.000 100.000 99.359 97.159 93.553
140 100.000 100.000 99.379 97.221 93.669
141 100.000 100.000 99.400 97.283 93.784
142 100.000 100.000 99.421 97.345 93.900
143 100.000 100.000 99.441 97.407 94.015
144 100.000 100.000 99.462 97.469 94.131
145 100.000 100.000 99.483 97.527 94.234
146 100.000 100.000 99.503 97.585 94.338
147 100.000 100.000 99.524 97.643 94.441
148 100.000 100.000 99.545 97.697 94.533
149 100.000 100.000 99.565 97.750 94.626
150 100.000 100.000 99.586 97.804 94.718
151 100.000 100.000 99.607 97.858 94.811
152 100.000 100.000 99.628 97.912 94.903
153 100.000 100.000 99.648 97.966 94.995
154 100.000 100.000 99.669 98.019 95.088
155 100.000 100.000 99.690 98.073 95.180
156 100.000 100.000 99.710 98.127 95.273
157 100.000 100.000 99.731 98.181 95.365
158 100.000 100.000 99.752 98.234 95.457
159 100.000 100.000 99.772 98.288 95.550
160 100.000 100.000 99.793 98.342 95.642
161 100.000 100.000 99.814 98.396 95.735
162 100.000 100.000 99.834 98.449 95.827
163 100.000 100.000 99.855 98.503 95.920
164 100.000 100.000 99.876 98.553 96.000
165 100.000 100.000 99.897 98.602 96.080
166 100.000 100.000 99.917 98.652 96.160
167 100.000 100.000 99.938 98.702 96.241
168 100.000 100.000 99.959 98.751 96.321
169 100.000 100.000 99.979 98.801 96.401
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 3, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 55.000 30.233 17.670 10.999 7.234
3 70.000 41.694 25.347 16.123 10.733
4 77.500 49.169 31.264 20.523 13.950
5 80.000 54.817 36.665 24.798 17.143
6 82.500 59.302 41.092 28.503 20.053
7 85.000 63.455 45.169 31.944 22.794
8 92.500 69.934 50.052 35.579 25.526
9 95.000 72.924 52.886 38.068 27.636
10 97.500 75.748 55.535 40.409 29.643
11 97.500 77.243 57.604 42.517 31.565
12 100.000 79.734 59.963 44.684 33.488
13 100.000 80.565 61.556 46.557 35.307
14 100.000 81.561 63.170 48.365 37.050
15 100.000 82.724 64.825 50.114 38.706
16 100.000 84.053 66.646 51.958 40.384
17 100.000 85.216 68.260 53.624 41.937
18 100.000 86.545 70.039 55.398 43.537
19 100.000 87.708 71.633 57.024 45.035
20 100.000 88.538 72.874 58.388 46.374
21 100.000 88.538 73.495 59.443 47.590
22 100.000 89.203 74.550 60.650 48.805
23 100.000 89.701 75.378 61.655 49.875
24 100.000 90.698 76.723 63.015 51.129
25 100.000 91.196 77.550 64.012 52.186
26 100.000 91.694 78.399 65.029 53.245
27 100.000 92.193 79.205 65.976 54.226
28 100.000 92.359 79.681 66.691 55.071
29 100.000 92.525 80.157 67.407 55.916
30 100.000 93.355 81.295 68.565 56.980
31 100.000 93.688 81.895 69.330 57.824
32 100.000 93.854 82.330 69.975 58.591
33 100.000 94.020 82.764 70.620 59.358
34 100.000 94.352 83.364 71.377 60.184
35 100.000 94.850 84.109 72.208 61.023
36 100.000 95.183 84.668 72.898 61.780
37 100.000 95.349 85.082 73.502 62.493
38 100.000 95.681 85.682 74.246 63.284
39 100.000 96.013 86.282 74.991 64.076
40 100.000 96.346 86.840 75.673 64.812
41 100.000 96.346 87.068 76.128 65.423
42 100.000 96.512 87.441 76.661 66.058
43 100.000 96.678 87.834 77.220 66.703
44 100.000 96.844 88.227 77.778 67.349
45 100.000 97.010 88.620 78.336 67.994
46 100.000 97.176 89.013 78.894 68.640
47 100.000 97.342 89.385 79.419 69.254
48 100.000 97.342 89.551 79.775 69.764
49 100.000 97.674 90.089 80.412 70.433
50 100.000 97.841 90.461 80.937 71.045
51 100.000 98.007 90.813 81.429 71.625
52 100.000 98.339 91.330 82.033 72.261
53 100.000 98.671 91.848 82.628 72.876
54 100.000 98.837 92.179 83.083 73.410
55 100.000 99.003 92.510 83.538 73.944
56 100.000 99.003 92.717 83.931 74.448
57 100.000 99.003 92.924 84.324 74.953
58 100.000 99.003 93.131 84.717 75.457
59 100.000 99.169 93.462 85.163 75.971
60 100.000 99.336 93.772 85.581 76.461
61 100.000 99.502 94.082 85.994 76.942
62 100.000 99.502 94.227 86.296 77.364
63 100.000 99.668 94.538 86.706 77.833
64 100.000 99.834 94.848 87.115 78.303
65 100.000 99.834 94.972 87.380 78.682
66 100.000 99.834 95.117 87.673 79.083
67 100.000 99.834 95.241 87.934 79.450
68 100.000 99.834 95.345 88.157 79.774
69 100.000 100.000 95.613 88.488 80.145
70 100.000 100.000 95.717 88.707 80.457
71 100.000 100.000 95.820 88.926 80.770
72 100.000 100.000 95.903 89.108 81.038
73 100.000 100.000 95.965 89.261 81.285
74 100.000 100.000 96.048 89.443 81.554
75 100.000 100.000 96.131 89.625 81.823
76 100.000 100.000 96.193 89.778 82.068
77 100.000 100.000 96.255 89.931 82.314
78 100.000 100.000 96.317 90.084 82.559
79 100.000 100.000 96.379 90.237 82.805
80 100.000 100.000 96.441 90.382 83.030
81 100.000 100.000 96.503 90.526 83.255
82 100.000 100.000 96.565 90.671 83.480
83 100.000 100.000 96.627 90.816 83.705
84 100.000 100.000 96.689 90.961 83.930
85 100.000 100.000 96.752 91.105 84.155
86 100.000 100.000 96.814 91.250 84.380
87 100.000 100.000 96.876 91.391 84.594
88 100.000 100.000 96.917 91.502 84.784
89 100.000 100.000 96.958 91.614 84.975
90 100.000 100.000 97.000 91.726 85.166
91 100.000 100.000 97.041 91.837 85.357
92 100.000 100.000 97.083 91.949 85.548
93 100.000 100.000 97.124 92.061 85.739
94 100.000 100.000 97.165 92.172 85.930
95 100.000 100.000 97.227 92.309 86.132
96 100.000 100.000 97.289 92.445 86.334
97 100.000 100.000 97.352 92.582 86.535
98 100.000 100.000 97.414 92.718 86.737
99 100.000 100.000 97.476 92.854 86.939
100 100.000 100.000 97.538 92.991 87.141
101 100.000 100.000 97.600 93.127 87.343
102 100.000 100.000 97.662 93.264 87.545
103 100.000 100.000 97.724 93.400 87.746
104 100.000 100.000 97.786 93.537 87.948
105 100.000 100.000 97.848 93.673 88.150
106 100.000 100.000 97.910 93.810 88.352
107 100.000 100.000 97.972 93.946 88.554
108 100.000 100.000 98.034 94.083 88.756
109 100.000 100.000 98.096 94.219 88.957
110 100.000 100.000 98.138 94.322 89.128
111 100.000 100.000 98.179 94.426 89.298
112 100.000 100.000 98.221 94.529 89.468
113 100.000 100.000 98.262 94.633 89.638
114 100.000 100.000 98.303 94.736 89.808
115 100.000 100.000 98.345 94.839 89.979
116 100.000 100.000 98.386 94.934 90.126
117 100.000 100.000 98.427 95.030 90.273
118 100.000 100.000 98.469 95.125 90.420
119 100.000 100.000 98.510 95.220 90.567
120 100.000 100.000 98.552 95.315 90.714
121 100.000 100.000 98.593 95.410 90.861
122 100.000 100.000 98.634 95.505 91.009
123 100.000 100.000 98.676 95.600 91.156
124 100.000 100.000 98.717 95.695 91.303
125 100.000 100.000 98.759 95.790 91.450
126 100.000 100.000 98.800 95.886 91.597
127 100.000 100.000 98.841 95.981 91.744
128 100.000 100.000 98.883 96.076 91.891
129 100.000 100.000 98.924 96.171 92.038
130 100.000 100.000 98.965 96.266 92.186
131 100.000 100.000 99.007 96.361 92.333
132 100.000 100.000 99.048 96.456 92.480
133 100.000 100.000 99.090 96.551 92.627
134 100.000 100.000 99.131 96.646 92.774
135 100.000 100.000 99.172 96.742 92.921
136 100.000 100.000 99.214 96.837 93.068
137 100.000 100.000 99.255 96.932 93.215
138 100.000 100.000 99.297 97.027 93.363
139 100.000 100.000 99.338 97.122 93.510
140 100.000 100.000 99.379 97.217 93.657
141 100.000 100.000 99.400 97.279 93.772
142 100.000 100.000 99.421 97.341 93.888
143 100.000 100.000 99.441 97.403 94.003
144 100.000 100.000 99.462 97.465 94.119
145 100.000 100.000 99.483 97.527 94.234
146 100.000 100.000 99.503 97.589 94.350
147 100.000 100.000 99.524 97.647 94.453
148 100.000 100.000 99.545 97.701 94.546
149 100.000 100.000 99.565 97.755 94.638
150 100.000 100.000 99.586 97.808 94.730
151 100.000 100.000 99.607 97.862 94.823
152 100.000 100.000 99.628 97.916 94.915
153 100.000 100.000 99.648 97.970 95.008
154 100.000 100.000 99.669 98.023 95.100
155 100.000 100.000 99.690 98.077 95.192
156 100.000 100.000 99.710 98.131 95.285
157 100.000 100.000 99.731 98.185 95.377
158 100.000 100.000 99.752 98.238 95.470
159 100.000 100.000 99.772 98.292 95.562
160 100.000 100.000 99.793 98.346 95.654
161 100.000 100.000 99.814 98.400 95.747
162 100.000 100.000 99.834 98.453 95.839
163 100.000 100.000 99.855 98.503 95.920
164 100.000 100.000 99.876 98.553 96.000
165 100.000 100.000 99.897 98.602 96.080
166 100.000 100.000 99.917 98.652 96.160
167 100.000 100.000 99.938 98.702 96.241
168 100.000 100.000 99.959 98.751 96.321
169 100.000 100.000 99.979 98.801 96.401
170 100.000 100.000 100.000 98.850 96.481

sed

Generation Strength: 4, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 77.500 54.153 36.582 24.819 17.161
6 85.000 60.631 41.486 28.487 19.927
7 90.000 65.781 45.810 31.923 22.608
8 92.500 69.767 49.638 35.128 25.169
9 95.000 72.924 52.638 37.733 27.346
10 97.500 75.748 55.535 40.421 29.669
11 100.000 78.571 58.101 42.613 31.500
12 100.000 80.066 60.170 44.721 33.422
13 100.000 81.561 62.239 46.822 35.326
14 100.000 83.056 64.329 48.948 37.248
15 100.000 84.219 65.901 50.536 38.703
16 100.000 85.382 67.474 52.123 40.157
17 100.000 86.379 68.777 53.397 41.289
18 100.000 87.375 70.122 54.757 42.542
19 100.000 88.372 71.591 56.329 44.032
20 100.000 89.203 72.771 57.574 45.217
21 100.000 90.033 73.909 58.731 46.281
22 100.000 90.864 75.460 60.551 48.056
23 100.000 91.362 76.309 61.572 49.126
24 100.000 91.860 77.136 62.552 50.142
25 100.000 92.359 77.985 63.569 51.201
26 100.000 92.857 78.854 64.628 52.314
27 100.000 93.355 79.764 65.769 53.536
28 100.000 93.688 80.364 66.522 54.351
29 100.000 94.020 81.006 67.357 55.275
30 100.000 94.352 81.523 67.953 55.890
31 100.000 94.684 82.164 68.792 56.824
32 100.000 95.017 82.744 69.499 57.572
33 100.000 95.349 83.344 70.252 58.387
34 100.000 95.681 83.944 71.004 59.201
35 100.000 95.847 84.378 71.650 59.968
36 100.000 96.013 84.813 72.295 60.736
37 100.000 96.179 85.082 72.617 61.083
38 100.000 96.346 85.433 73.122 61.695
39 100.000 96.512 85.806 73.655 62.330
40 100.000 96.678 86.220 74.259 63.042
41 100.000 96.844 86.551 74.705 63.555
42 100.000 97.010 86.903 75.193 64.123
43 100.000 97.176 87.172 75.516 64.471
44 100.000 97.342 87.544 76.041 65.082
45 100.000 97.508 87.896 76.521 65.630
46 100.000 97.674 88.206 76.930 66.099
47 100.000 97.841 88.496 77.298 66.513
48 100.000 98.007 88.785 77.666 66.928
49 100.000 98.173 89.137 78.154 67.496
50 100.000 98.339 89.406 78.477 67.844
51 100.000 98.505 89.799 79.051 68.533
52 100.000 98.671 90.130 79.498 69.046
53 100.000 98.837 90.399 79.821 69.394
54 100.000 99.003 90.710 80.230 69.863
55 100.000 99.169 91.020 80.639 70.333
56 100.000 99.336 91.372 81.127 70.900
57 100.000 99.502 91.641 81.450 71.248
58 100.000 99.668 91.930 81.818 71.663
59 100.000 99.834 92.220 82.186 72.077
60 100.000 100.000 92.489 82.508 72.425
61 100.000 100.000 92.551 82.649 72.639
62 100.000 100.000 92.572 82.707 72.742
63 100.000 100.000 92.593 82.756 72.823
64 100.000 100.000 92.717 83.025 73.213
65 100.000 100.000 92.800 83.199 73.459
66 100.000 100.000 92.903 83.418 73.771
67 100.000 100.000 93.048 83.720 74.193
68 100.000 100.000 93.131 83.910 74.485
69 100.000 100.000 93.172 84.005 74.632
70 100.000 100.000 93.213 84.100 74.779
71 100.000 100.000 93.358 84.402 75.201
72 100.000 100.000 93.462 84.626 75.524
73 100.000 100.000 93.524 84.779 75.770
74 100.000 100.000 93.524 84.787 75.795
75 100.000 100.000 93.544 84.845 75.899
76 100.000 100.000 93.606 84.985 76.113
77 100.000 100.000 93.627 85.035 76.193
78 100.000 100.000 93.669 85.138 76.361
79 100.000 100.000 93.731 85.279 76.575
80 100.000 100.000 93.855 85.539 76.942
81 100.000 100.000 93.938 85.721 77.211
82 100.000 100.000 94.041 85.945 77.534
83 100.000 100.000 94.082 86.040 77.681
84 100.000 100.000 94.165 86.222 77.950
85 100.000 100.000 94.248 86.404 78.219
86 100.000 100.000 94.289 86.507 78.389
87 100.000 100.000 94.331 86.602 78.536
88 100.000 100.000 94.351 86.664 78.652
89 100.000 100.000 94.455 86.887 78.975
90 100.000 100.000 94.475 86.945 79.078
91 100.000 100.000 94.558 87.127 79.347
92 100.000 100.000 94.558 87.131 79.360
93 100.000 100.000 94.641 87.313 79.629
94 100.000 100.000 94.641 87.318 79.643
95 100.000 100.000 94.682 87.429 79.833
96 100.000 100.000 94.703 87.479 79.914
97 100.000 100.000 94.786 87.669 80.203
98 100.000 100.000 94.848 87.805 80.405
99 100.000 100.000 94.889 87.901 80.552
100 100.000 100.000 94.931 87.996 80.699
101 100.000 100.000 95.055 88.256 81.066
102 100.000 100.000 95.096 88.360 81.237
103 100.000 100.000 95.096 88.364 81.250
104 100.000 100.000 95.158 88.500 81.452
105 100.000 100.000 95.158 88.525 81.521
106 100.000 100.000 95.220 88.661 81.723
107 100.000 100.000 95.282 88.798 81.925
108 100.000 100.000 95.427 89.100 82.347
109 100.000 100.000 95.427 89.116 82.395
110 100.000 100.000 95.551 89.377 82.762
111 100.000 100.000 95.613 89.513 82.964
112 100.000 100.000 95.634 89.563 83.045
113 100.000 100.000 95.676 89.658 83.192
114 100.000 100.000 95.820 89.960 83.614
115 100.000 100.000 95.841 90.010 83.694
116 100.000 100.000 95.882 90.121 83.885
117 100.000 100.000 95.924 90.216 84.032
118 100.000 100.000 95.945 90.266 84.112
119 100.000 100.000 96.007 90.402 84.314
120 100.000 100.000 96.027 90.469 84.438
121 100.000 100.000 96.048 90.518 84.518
122 100.000 100.000 96.089 90.613 84.665
123 100.000 100.000 96.172 90.812 84.978
124 100.000 100.000 96.214 90.907 85.125
125 100.000 100.000 96.214 90.915 85.150
126 100.000 100.000 96.234 90.973 85.254
127 100.000 100.000 96.255 91.035 85.369
128 100.000 100.000 96.255 91.043 85.395
129 100.000 100.000 96.296 91.138 85.542
130 100.000 100.000 96.317 91.188 85.622
131 100.000 100.000 96.379 91.329 85.836
132 100.000 100.000 96.379 91.341 85.873
133 100.000 100.000 96.379 91.349 85.898
134 100.000 100.000 96.400 91.399 85.978
135 100.000 100.000 96.441 91.494 86.126
136 100.000 100.000 96.483 91.593 86.285
137 100.000 100.000 96.483 91.597 86.298
138 100.000 100.000 96.503 91.647 86.379
139 100.000 100.000 96.545 91.742 86.526
140 100.000 100.000 96.586 91.837 86.673
141 100.000 100.000 96.607 91.887 86.753
142 100.000 100.000 96.648 91.982 86.900
143 100.000 100.000 96.648 91.986 86.913
144 100.000 100.000 96.648 91.994 86.939
145 100.000 100.000 96.689 92.089 87.086
146 100.000 100.000 96.689 92.098 87.112
147 100.000 100.000 96.689 92.106 87.137
148 100.000 100.000 96.689 92.114 87.163
149 100.000 100.000 96.731 92.209 87.310
150 100.000 100.000 96.772 92.305 87.457
151 100.000 100.000 96.814 92.416 87.648
152 100.000 100.000 96.855 92.511 87.795
153 100.000 100.000 96.855 92.520 87.821
154 100.000 100.000 96.876 92.586 87.945
155 100.000 100.000 96.896 92.635 88.025
156 100.000 100.000 96.896 92.644 88.050
157 100.000 100.000 96.896 92.648 88.064
158 100.000 100.000 96.938 92.751 88.234
159 100.000 100.000 97.000 92.888 88.436
160 100.000 100.000 97.621 93.942 89.652
161 100.000 100.000 97.683 94.078 89.853
162 100.000 100.000 97.683 94.083 89.867
163 100.000 100.000 97.683 94.087 89.880
164 100.000 100.000 97.724 94.182 90.027
165 100.000 100.000 97.724 94.190 90.053
166 100.000 100.000 97.724 94.203 90.089
167 100.000 100.000 97.724 94.211 90.115
168 100.000 100.000 97.724 94.219 90.140
169 100.000 100.000 97.765 94.314 90.288
170 100.000 100.000 97.786 94.364 90.368
171 100.000 100.000 97.786 94.372 90.393
172 100.000 100.000 97.827 94.467 90.540
173 100.000 100.000 97.848 94.521 90.633
174 100.000 100.000 97.848 94.525 90.646
175 100.000 100.000 97.931 94.707 90.915
176 100.000 100.000 97.952 94.757 90.995
177 100.000 100.000 97.972 94.806 91.075
178 100.000 100.000 97.993 94.856 91.156
179 100.000 100.000 97.993 94.872 91.204
180 100.000 100.000 98.034 94.968 91.351
181 100.000 100.000 98.034 94.980 91.388
182 100.000 100.000 98.055 95.030 91.468
183 100.000 100.000 98.076 95.083 91.561
184 100.000 100.000 98.117 95.178 91.708
185 100.000 100.000 98.158 95.274 91.855
186 100.000 100.000 98.200 95.369 92.002
187 100.000 100.000 98.200 95.373 92.015
188 100.000 100.000 98.221 95.439 92.139
189 100.000 100.000 98.241 95.489 92.220
190 100.000 100.000 98.241 95.505 92.268
191 100.000 100.000 98.262 95.555 92.348
192 100.000 100.000 98.262 95.559 92.362
193 100.000 100.000 98.262 95.563 92.375
194 100.000 100.000 98.303 95.658 92.522
195 100.000 100.000 98.303 95.671 92.559
196 100.000 100.000 98.303 95.675 92.572
197 100.000 100.000 98.365 95.811 92.774
198 100.000 100.000 98.386 95.861 92.854
199 100.000 100.000 98.448 95.997 93.056
200 100.000 100.000 98.448 96.001 93.069
201 100.000 100.000 98.448 96.010 93.095
202 100.000 100.000 98.448 96.014 93.108
203 100.000 100.000 98.448 96.039 93.178
204 100.000 100.000 98.490 96.134 93.325
205 100.000 100.000 98.531 96.229 93.472
206 100.000 100.000 98.531 96.237 93.497
207 100.000 100.000 98.531 96.245 93.523
208 100.000 100.000 98.552 96.295 93.603
209 100.000 100.000 98.572 96.345 93.684
210 100.000 100.000 98.572 96.357 93.720
211 100.000 100.000 98.572 96.365 93.746
212 100.000 100.000 98.572 96.373 93.771
213 100.000 100.000 98.634 96.510 93.973
214 100.000 100.000 98.634 96.522 94.009
215 100.000 100.000 98.634 96.531 94.035
216 100.000 100.000 98.655 96.580 94.115
217 100.000 100.000 98.676 96.630 94.195
218 100.000 100.000 98.696 96.679 94.276
219 100.000 100.000 98.696 96.688 94.301
220 100.000 100.000 98.696 96.692 94.315
221 100.000 100.000 98.696 96.696 94.328
222 100.000 100.000 98.717 96.750 94.420
223 100.000 100.000 98.738 96.804 94.513
224 100.000 100.000 98.738 96.808 94.526
225 100.000 100.000 98.759 96.857 94.606
226 100.000 100.000 98.759 96.866 94.632
227 100.000 100.000 98.759 96.870 94.645
228 100.000 100.000 98.800 96.965 94.792
229 100.000 100.000 98.821 97.014 94.873
230 100.000 100.000 98.821 97.019 94.886
231 100.000 100.000 98.821 97.027 94.912
232 100.000 100.000 98.821 97.039 94.948
233 100.000 100.000 98.821 97.043 94.961
234 100.000 100.000 98.862 97.138 95.109
235 100.000 100.000 98.862 97.143 95.122
236 100.000 100.000 98.903 97.238 95.269
237 100.000 100.000 98.945 97.333 95.416
238 100.000 100.000 98.986 97.428 95.563
239 100.000 100.000 98.986 97.436 95.589
240 100.000 100.000 99.028 97.531 95.736
241 100.000 100.000 99.069 97.626 95.883
242 100.000 100.000 99.069 97.635 95.909
243 100.000 100.000 99.069 97.643 95.934
244 100.000 100.000 99.110 97.738 96.081
245 100.000 100.000 99.110 97.746 96.107
246 100.000 100.000 99.110 97.755 96.132
247 100.000 100.000 99.110 97.763 96.158
248 100.000 100.000 99.110 97.767 96.171
249 100.000 100.000 99.110 97.771 96.185
250 100.000 100.000 99.110 97.775 96.198
251 100.000 100.000 99.152 97.870 96.345
252 100.000 100.000 99.152 97.875 96.358
253 100.000 100.000 99.152 97.879 96.372
254 100.000 100.000 99.152 97.887 96.397
255 100.000 100.000 99.152 97.891 96.411
256 100.000 100.000 99.152 97.899 96.436
257 100.000 100.000 99.152 97.908 96.462
258 100.000 100.000 99.172 97.957 96.542
259 100.000 100.000 99.379 98.350 97.047
260 100.000 100.000 99.379 98.358 97.072
261 100.000 100.000 99.379 98.367 97.098
262 100.000 100.000 99.379 98.371 97.111
263 100.000 100.000 99.379 98.375 97.124
264 100.000 100.000 99.379 98.391 97.173
265 100.000 100.000 99.379 98.400 97.199
266 100.000 100.000 99.421 98.495 97.346
267 100.000 100.000 99.441 98.549 97.438
268 100.000 100.000 99.441 98.553 97.452
269 100.000 100.000 99.462 98.602 97.532
270 100.000 100.000 99.483 98.652 97.612
271 100.000 100.000 99.483 98.656 97.625
272 100.000 100.000 99.483 98.664 97.651
273 100.000 100.000 99.483 98.668 97.664
274 100.000 100.000 99.483 98.677 97.690
275 100.000 100.000 99.483 98.693 97.738
276 100.000 100.000 99.483 98.697 97.752
277 100.000 100.000 99.483 98.702 97.765
278 100.000 100.000 99.483 98.710 97.791
279 100.000 100.000 99.483 98.726 97.839
280 100.000 100.000 99.483 98.735 97.865
281 100.000 100.000 99.483 98.739 97.878
282 100.000 100.000 99.483 98.743 97.892
283 100.000 100.000 99.483 98.747 97.905
284 100.000 100.000 99.690 99.140 98.410
285 100.000 100.000 99.690 99.144 98.423
286 100.000 100.000 99.897 99.537 98.928
287 100.000 100.000 99.917 99.591 99.020
288 100.000 100.000 99.938 99.640 99.100
289 100.000 100.000 99.959 99.690 99.180
290 100.000 100.000 99.959 99.702 99.217
291 100.000 100.000 99.959 99.711 99.243
292 100.000 100.000 99.959 99.715 99.256
293 100.000 100.000 99.959 99.723 99.281
294 100.000 100.000 99.959 99.731 99.307
295 100.000 100.000 99.959 99.739 99.332
296 100.000 100.000 99.959 99.744 99.346
297 100.000 100.000 99.959 99.752 99.371
298 100.000 100.000 99.959 99.756 99.385
299 100.000 100.000 99.959 99.764 99.410
300 100.000 100.000 99.959 99.773 99.436
301 100.000 100.000 99.959 99.777 99.449
302 100.000 100.000 99.959 99.781 99.463
303 100.000 100.000 99.959 99.785 99.476
304 100.000 100.000 99.959 99.789 99.489
305 100.000 100.000 99.959 99.793 99.503
306 100.000 100.000 99.959 99.802 99.528
307 100.000 100.000 99.959 99.806 99.542
308 100.000 100.000 99.959 99.814 99.567
309 100.000 100.000 99.959 99.822 99.593
310 100.000 100.000 99.959 99.830 99.618
311 100.000 100.000 100.000 99.926 99.765
312 100.000 100.000 100.000 99.930 99.779
313 100.000 100.000 100.000 99.938 99.804
314 100.000 100.000 100.000 99.946 99.830
315 100.000 100.000 100.000 99.950 99.843
316 100.000 100.000 100.000 99.955 99.857
317 100.000 100.000 100.000 99.959 99.870
318 100.000 100.000 100.000 99.963 99.883
319 100.000 100.000 100.000 99.967 99.897
320 100.000 100.000 100.000 99.971 99.910
321 100.000 100.000 100.000 99.979 99.936
322 100.000 100.000 100.000 99.988 99.961
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 4, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.834 31.761 20.775 14.053
5 82.500 56.645 37.596 25.195 17.289
6 87.500 62.458 42.541 29.045 20.207
7 90.000 66.279 46.224 32.151 22.699
8 92.500 69.934 49.721 35.107 25.090
9 95.000 72.757 52.617 37.795 27.412
10 97.500 75.581 55.369 40.284 29.557
11 100.000 78.405 58.018 42.625 31.564
12 100.000 79.900 60.066 44.697 33.443
13 100.000 81.395 62.094 46.735 35.287
14 100.000 82.724 63.977 48.687 37.089
15 100.000 83.555 65.570 50.560 38.908
16 100.000 84.884 67.349 52.334 40.508
17 100.000 86.047 68.943 53.959 42.006
18 100.000 87.043 70.412 55.531 43.496
19 100.000 88.040 71.860 57.081 44.980
20 100.000 89.037 73.246 58.512 46.312
21 100.000 90.033 74.591 59.897 47.620
22 100.000 90.864 75.729 61.055 48.684
23 100.000 91.362 76.640 62.192 49.896
24 100.000 91.362 77.261 63.247 51.112
25 100.000 91.860 78.109 64.264 52.171
26 100.000 92.359 78.957 65.286 53.241
27 100.000 92.857 79.785 66.266 54.256
28 100.000 93.355 80.612 67.246 55.271
29 100.000 93.688 81.295 68.147 56.262
30 100.000 94.020 81.937 68.995 57.217
31 100.000 94.352 82.557 69.785 58.075
32 100.000 94.684 83.178 70.574 58.934
33 100.000 95.017 83.799 71.368 59.803
34 100.000 95.349 84.399 72.121 60.618
35 100.000 95.681 84.978 72.828 61.365
36 100.000 96.013 85.537 73.502 62.079
37 100.000 96.346 86.075 74.139 62.749
38 100.000 96.512 86.509 74.767 63.473
39 100.000 96.678 86.944 75.412 64.240
40 100.000 96.844 87.378 76.058 65.007
41 100.000 97.010 87.792 76.665 65.730
42 100.000 97.176 88.206 77.269 66.443
43 100.000 97.342 88.599 77.827 67.089
44 100.000 97.508 88.992 78.386 67.734
45 100.000 97.674 89.385 78.944 68.380
46 100.000 97.841 89.779 79.502 69.025
47 100.000 98.007 90.172 80.069 69.694
48 100.000 98.173 90.544 80.602 70.329
49 100.000 98.339 90.896 81.090 70.897
50 100.000 98.505 91.248 81.590 71.497
51 100.000 98.671 91.599 82.078 72.065
52 100.000 98.837 91.930 82.533 72.599
53 100.000 99.003 92.262 82.988 73.133
54 100.000 99.169 92.593 83.443 73.666
55 100.000 99.336 92.924 83.898 74.200
56 100.000 99.502 93.255 84.361 74.757
57 100.000 99.668 93.586 84.820 75.303
58 100.000 99.834 93.917 85.267 75.816
59 100.000 100.000 94.248 85.713 76.329
60 100.000 100.000 94.455 86.106 76.834
61 100.000 100.000 94.662 86.499 77.338
62 100.000 100.000 94.869 86.892 77.843
63 100.000 100.000 94.993 87.152 78.210
64 100.000 100.000 95.117 87.413 78.577
65 100.000 100.000 95.241 87.673 78.945
66 100.000 100.000 95.365 87.942 79.335
67 100.000 100.000 95.489 88.202 79.702
68 100.000 100.000 95.613 88.480 80.113
69 100.000 100.000 95.738 88.740 80.480
70 100.000 100.000 95.841 88.963 80.804
71 100.000 100.000 95.945 89.187 81.127
72 100.000 100.000 96.027 89.369 81.396
73 100.000 100.000 96.110 89.551 81.665
74 100.000 100.000 96.193 89.741 81.956
75 100.000 100.000 96.255 89.894 82.202
76 100.000 100.000 96.317 90.047 82.448
77 100.000 100.000 96.379 90.183 82.649
78 100.000 100.000 96.441 90.320 82.851
79 100.000 100.000 96.503 90.456 83.053
80 100.000 100.000 96.565 90.601 83.278
81 100.000 100.000 96.627 90.754 83.524
82 100.000 100.000 96.689 90.890 83.725
83 100.000 100.000 96.752 91.043 83.971
84 100.000 100.000 96.814 91.184 84.185
85 100.000 100.000 96.876 91.320 84.387
86 100.000 100.000 96.938 91.457 84.589
87 100.000 100.000 97.000 91.593 84.791
88 100.000 100.000 97.062 91.730 84.992
89 100.000 100.000 97.124 91.866 85.194
90 100.000 100.000 97.186 92.003 85.396
91 100.000 100.000 97.248 92.139 85.598
92 100.000 100.000 97.310 92.276 85.800
93 100.000 100.000 97.372 92.412 86.002
94 100.000 100.000 97.434 92.548 86.203
95 100.000 100.000 97.496 92.685 86.405
96 100.000 100.000 97.558 92.821 86.607
97 100.000 100.000 97.621 92.958 86.809
98 100.000 100.000 97.683 93.094 87.011
99 100.000 100.000 97.745 93.247 87.256
100 100.000 100.000 97.807 93.400 87.502
101 100.000 100.000 97.869 93.545 87.727
102 100.000 100.000 97.931 93.682 87.929
103 100.000 100.000 97.993 93.818 88.131
104 100.000 100.000 98.055 93.971 88.376
105 100.000 100.000 98.117 94.107 88.578
106 100.000 100.000 98.179 94.260 88.824
107 100.000 100.000 98.241 94.397 89.025
108 100.000 100.000 98.303 94.542 89.250
109 100.000 100.000 98.365 94.678 89.452
110 100.000 100.000 98.427 94.815 89.654
111 100.000 100.000 98.490 94.951 89.856
112 100.000 100.000 98.552 95.104 90.102
113 100.000 100.000 98.614 95.257 90.347
114 100.000 100.000 98.676 95.393 90.549
115 100.000 100.000 98.738 95.546 90.795
116 100.000 100.000 98.800 95.699 91.040
117 100.000 100.000 98.841 95.803 91.210
118 100.000 100.000 98.883 95.906 91.381
119 100.000 100.000 98.924 96.010 91.551
120 100.000 100.000 98.965 96.105 91.698
121 100.000 100.000 98.986 96.163 91.801
122 100.000 100.000 99.007 96.216 91.894
123 100.000 100.000 99.028 96.266 91.974
124 100.000 100.000 99.048 96.316 92.054
125 100.000 100.000 99.069 96.369 92.147
126 100.000 100.000 99.090 96.419 92.227
127 100.000 100.000 99.110 96.473 92.319
128 100.000 100.000 99.131 96.526 92.412
129 100.000 100.000 99.152 96.584 92.515
130 100.000 100.000 99.172 96.638 92.607
131 100.000 100.000 99.193 96.688 92.688
132 100.000 100.000 99.214 96.746 92.791
133 100.000 100.000 99.234 96.795 92.871
134 100.000 100.000 99.255 96.849 92.964
135 100.000 100.000 99.276 96.899 93.044
136 100.000 100.000 99.297 96.952 93.136
137 100.000 100.000 99.317 97.006 93.229
138 100.000 100.000 99.338 97.068 93.344
139 100.000 100.000 99.359 97.122 93.437
140 100.000 100.000 99.379 97.172 93.517
141 100.000 100.000 99.400 97.242 93.653
142 100.000 100.000 99.421 97.300 93.756
143 100.000 100.000 99.441 97.354 93.849
144 100.000 100.000 99.462 97.407 93.941
145 100.000 100.000 99.483 97.461 94.034
146 100.000 100.000 99.503 97.511 94.114
147 100.000 100.000 99.524 97.569 94.217
148 100.000 100.000 99.545 97.626 94.321
149 100.000 100.000 99.565 97.676 94.401
150 100.000 100.000 99.586 97.734 94.504
151 100.000 100.000 99.607 97.784 94.584
152 100.000 100.000 99.628 97.837 94.677
153 100.000 100.000 99.648 97.887 94.757
154 100.000 100.000 99.669 97.937 94.837
155 100.000 100.000 99.690 97.990 94.930
156 100.000 100.000 99.710 98.044 95.022
157 100.000 100.000 99.731 98.102 95.126
158 100.000 100.000 99.752 98.160 95.229
159 100.000 100.000 99.772 98.214 95.321
160 100.000 100.000 99.793 98.267 95.414
161 100.000 100.000 99.814 98.325 95.517
162 100.000 100.000 99.834 98.379 95.609
163 100.000 100.000 99.855 98.437 95.713
164 100.000 100.000 99.876 98.491 95.805
165 100.000 100.000 99.897 98.544 95.898
166 100.000 100.000 99.917 98.598 95.990
167 100.000 100.000 99.938 98.656 96.093
168 100.000 100.000 99.959 98.710 96.186
169 100.000 100.000 99.979 98.764 96.278
170 100.000 100.000 100.000 98.813 96.358
171 100.000 100.000 100.000 98.830 96.407
172 100.000 100.000 100.000 98.838 96.433
173 100.000 100.000 100.000 98.846 96.458
174 100.000 100.000 100.000 98.859 96.495
175 100.000 100.000 100.000 98.875 96.543
176 100.000 100.000 100.000 98.884 96.569
177 100.000 100.000 100.000 98.892 96.594
178 100.000 100.000 100.000 98.900 96.620
179 100.000 100.000 100.000 98.908 96.645
180 100.000 100.000 100.000 98.917 96.671
181 100.000 100.000 100.000 98.925 96.696
182 100.000 100.000 100.000 98.941 96.745
183 100.000 100.000 100.000 98.950 96.771
184 100.000 100.000 100.000 98.958 96.796
185 100.000 100.000 100.000 98.970 96.833
186 100.000 100.000 100.000 98.979 96.858
187 100.000 100.000 100.000 99.003 96.927
188 100.000 100.000 100.000 99.008 96.941
189 100.000 100.000 100.000 99.012 96.954
190 100.000 100.000 100.000 99.020 96.980
191 100.000 100.000 100.000 99.024 96.993
192 100.000 100.000 100.000 99.049 97.062
193 100.000 100.000 100.000 99.065 97.111
194 100.000 100.000 100.000 99.070 97.124
195 100.000 100.000 100.000 99.086 97.173
196 100.000 100.000 100.000 99.090 97.186
197 100.000 100.000 100.000 99.099 97.212
198 100.000 100.000 100.000 99.107 97.238
199 100.000 100.000 100.000 99.111 97.251
200 100.000 100.000 100.000 99.115 97.264
201 100.000 100.000 100.000 99.123 97.290
202 100.000 100.000 100.000 99.148 97.359
203 100.000 100.000 100.000 99.156 97.385
204 100.000 100.000 100.000 99.165 97.410
205 100.000 100.000 100.000 99.169 97.424
206 100.000 100.000 100.000 99.177 97.449
207 100.000 100.000 100.000 99.194 97.498
208 100.000 100.000 100.000 99.198 97.511
209 100.000 100.000 100.000 99.206 97.537
210 100.000 100.000 100.000 99.214 97.562
211 100.000 100.000 100.000 99.223 97.588
212 100.000 100.000 100.000 99.227 97.601
213 100.000 100.000 100.000 99.235 97.627
214 100.000 100.000 100.000 99.252 97.675
215 100.000 100.000 100.000 99.256 97.689
216 100.000 100.000 100.000 99.260 97.702
217 100.000 100.000 100.000 99.268 97.728
218 100.000 100.000 100.000 99.276 97.753
219 100.000 100.000 100.000 99.280 97.766
220 100.000 100.000 100.000 99.289 97.792
221 100.000 100.000 100.000 99.301 97.828
222 100.000 100.000 100.000 99.309 97.854
223 100.000 100.000 100.000 99.318 97.880
224 100.000 100.000 100.000 99.326 97.905
225 100.000 100.000 100.000 99.334 97.931
226 100.000 100.000 100.000 99.338 97.944
227 100.000 100.000 100.000 99.343 97.957
228 100.000 100.000 100.000 99.355 97.994
229 100.000 100.000 100.000 99.363 98.019
230 100.000 100.000 100.000 99.367 98.033
231 100.000 100.000 100.000 99.371 98.046
232 100.000 100.000 100.000 99.384 98.083
233 100.000 100.000 100.000 99.392 98.108
234 100.000 100.000 100.000 99.409 98.157
235 100.000 100.000 100.000 99.417 98.182
236 100.000 100.000 100.000 99.425 98.208
237 100.000 100.000 100.000 99.433 98.233
238 100.000 100.000 100.000 99.442 98.259
239 100.000 100.000 100.000 99.450 98.284
240 100.000 100.000 100.000 99.454 98.298
241 100.000 100.000 100.000 99.462 98.323
242 100.000 100.000 100.000 99.467 98.337
243 100.000 100.000 100.000 99.475 98.362
244 100.000 100.000 100.000 99.483 98.388
245 100.000 100.000 100.000 99.487 98.401
246 100.000 100.000 100.000 99.491 98.414
247 100.000 100.000 100.000 99.496 98.428
248 100.000 100.000 100.000 99.500 98.441
249 100.000 100.000 100.000 99.508 98.467
250 100.000 100.000 100.000 99.512 98.480
251 100.000 100.000 100.000 99.516 98.494
252 100.000 100.000 100.000 99.529 98.530
253 100.000 100.000 100.000 99.533 98.543
254 100.000 100.000 100.000 99.549 98.592
255 100.000 100.000 100.000 99.558 98.618
256 100.000 100.000 100.000 99.566 98.643
257 100.000 100.000 100.000 99.582 98.692
258 100.000 100.000 100.000 99.586 98.705
259 100.000 100.000 100.000 99.595 98.731
260 100.000 100.000 100.000 99.603 98.756
261 100.000 100.000 100.000 99.611 98.782
262 100.000 100.000 100.000 99.620 98.807
263 100.000 100.000 100.000 99.628 98.833
264 100.000 100.000 100.000 99.632 98.846
265 100.000 100.000 100.000 99.640 98.872
266 100.000 100.000 100.000 99.657 98.920
267 100.000 100.000 100.000 99.661 98.934
268 100.000 100.000 100.000 99.669 98.959
269 100.000 100.000 100.000 99.677 98.985
270 100.000 100.000 100.000 99.682 98.998
271 100.000 100.000 100.000 99.690 99.024
272 100.000 100.000 100.000 99.698 99.049
273 100.000 100.000 100.000 99.702 99.063
274 100.000 100.000 100.000 99.711 99.088
275 100.000 100.000 100.000 99.719 99.114
276 100.000 100.000 100.000 99.723 99.127
277 100.000 100.000 100.000 99.727 99.140
278 100.000 100.000 100.000 99.735 99.166
279 100.000 100.000 100.000 99.744 99.191
280 100.000 100.000 100.000 99.752 99.217
281 100.000 100.000 100.000 99.760 99.243
282 100.000 100.000 100.000 99.764 99.256
283 100.000 100.000 100.000 99.773 99.281
284 100.000 100.000 100.000 99.781 99.307
285 100.000 100.000 100.000 99.785 99.320
286 100.000 100.000 100.000 99.789 99.334
287 100.000 100.000 100.000 99.793 99.347
288 100.000 100.000 100.000 99.797 99.360
289 100.000 100.000 100.000 99.802 99.374
290 100.000 100.000 100.000 99.806 99.387
291 100.000 100.000 100.000 99.810 99.401
292 100.000 100.000 100.000 99.818 99.426
293 100.000 100.000 100.000 99.822 99.439
294 100.000 100.000 100.000 99.826 99.453
295 100.000 100.000 100.000 99.835 99.478
296 100.000 100.000 100.000 99.839 99.492
297 100.000 100.000 100.000 99.843 99.505
298 100.000 100.000 100.000 99.847 99.519
299 100.000 100.000 100.000 99.855 99.544
300 100.000 100.000 100.000 99.859 99.557
301 100.000 100.000 100.000 99.864 99.571
302 100.000 100.000 100.000 99.868 99.584
303 100.000 100.000 100.000 99.872 99.598
304 100.000 100.000 100.000 99.876 99.611
305 100.000 100.000 100.000 99.880 99.624
306 100.000 100.000 100.000 99.884 99.638
307 100.000 100.000 100.000 99.892 99.663
308 100.000 100.000 100.000 99.901 99.689
309 100.000 100.000 100.000 99.909 99.714
310 100.000 100.000 100.000 99.913 99.728
311 100.000 100.000 100.000 99.921 99.753
312 100.000 100.000 100.000 99.926 99.767
313 100.000 100.000 100.000 99.930 99.780
314 100.000 100.000 100.000 99.934 99.793
315 100.000 100.000 100.000 99.942 99.819
316 100.000 100.000 100.000 99.946 99.832
317 100.000 100.000 100.000 99.955 99.858
318 100.000 100.000 100.000 99.963 99.883
319 100.000 100.000 100.000 99.967 99.897
320 100.000 100.000 100.000 99.971 99.910
321 100.000 100.000 100.000 99.979 99.936
322 100.000 100.000 100.000 99.988 99.961
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 4, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 43.023 25.926 16.354 10.818
4 75.000 48.671 31.326 20.630 14.011
5 80.000 54.817 36.623 24.745 17.099
6 87.500 61.794 42.044 28.793 20.103
7 90.000 66.113 46.286 32.349 22.912
8 92.500 69.934 50.010 35.525 25.482
9 95.000 72.591 52.617 37.927 27.585
10 95.000 74.252 54.893 40.218 29.643
11 95.000 75.415 56.756 42.240 31.529
12 97.500 78.239 59.363 44.511 33.459
13 100.000 80.731 61.701 46.648 35.355
14 100.000 82.060 63.584 48.600 37.157
15 100.000 83.555 65.570 50.560 38.903
16 100.000 84.718 67.184 52.227 40.456
17 100.000 85.880 68.777 53.852 41.954
18 100.000 86.877 70.226 55.403 43.439
19 100.000 87.874 71.633 56.875 44.825
20 100.000 88.870 73.019 58.314 46.177
21 100.000 89.867 74.405 59.744 47.508
22 100.000 89.867 75.026 60.799 48.724
23 100.000 90.365 75.916 61.899 49.892
24 100.000 90.864 76.785 62.957 51.006
25 100.000 91.362 77.654 64.016 52.120
26 100.000 91.860 78.502 65.033 53.179
27 100.000 92.691 79.640 66.191 54.243
28 100.000 93.189 80.468 67.171 55.258
29 100.000 93.688 81.295 68.151 56.273
30 100.000 94.020 81.916 68.945 57.143
31 100.000 94.352 82.537 69.735 58.001
32 100.000 94.684 83.157 70.525 58.860
33 100.000 95.017 83.758 71.294 59.715
34 100.000 95.349 84.358 72.046 60.530
35 100.000 95.515 84.792 72.692 61.297
36 100.000 95.681 85.227 73.337 62.065
37 100.000 95.847 85.661 73.982 62.832
38 100.000 96.013 86.096 74.623 63.588
39 100.000 96.179 86.530 75.259 64.335
40 100.000 96.346 86.965 75.888 65.058
41 100.000 96.512 87.378 76.492 65.771
42 100.000 96.844 87.916 77.141 66.473
43 100.000 97.176 88.454 77.778 67.143
44 100.000 97.342 88.848 78.336 67.789
45 100.000 97.508 89.241 78.894 68.435
46 100.000 97.674 89.634 79.453 69.080
47 100.000 97.841 90.027 80.011 69.726
48 100.000 98.173 90.544 80.614 70.362
49 100.000 98.339 90.896 81.102 70.930
50 100.000 98.505 91.248 81.590 71.497
51 100.000 98.671 91.599 82.078 72.065
52 100.000 98.837 91.930 82.533 72.599
53 100.000 99.003 92.262 82.988 73.133
54 100.000 99.169 92.593 83.443 73.666
55 100.000 99.336 92.924 83.898 74.200
56 100.000 99.502 93.255 84.353 74.734
57 100.000 99.668 93.586 84.808 75.268
58 100.000 99.668 93.793 85.200 75.772
59 100.000 99.668 94.000 85.593 76.277
60 100.000 99.668 94.206 85.986 76.782
61 100.000 99.834 94.538 86.433 77.295
62 100.000 100.000 94.869 86.879 77.808
63 100.000 100.000 95.013 87.189 78.253
64 100.000 100.000 95.158 87.495 78.687
65 100.000 100.000 95.282 87.772 79.098
66 100.000 100.000 95.407 88.033 79.465
67 100.000 100.000 95.531 88.293 79.832
68 100.000 100.000 95.655 88.554 80.199
69 100.000 100.000 95.758 88.777 80.523
70 100.000 100.000 95.862 89.001 80.846
71 100.000 100.000 95.965 89.224 81.170
72 100.000 100.000 96.048 89.406 81.438
73 100.000 100.000 96.131 89.588 81.707
74 100.000 100.000 96.193 89.741 81.953
75 100.000 100.000 96.255 89.894 82.198
76 100.000 100.000 96.317 90.047 82.444
77 100.000 100.000 96.379 90.200 82.690
78 100.000 100.000 96.441 90.353 82.935
79 100.000 100.000 96.503 90.506 83.181
80 100.000 100.000 96.565 90.659 83.426
81 100.000 100.000 96.627 90.812 83.672
82 100.000 100.000 96.689 90.965 83.918
83 100.000 100.000 96.752 91.118 84.163
84 100.000 100.000 96.814 91.271 84.409
85 100.000 100.000 96.876 91.424 84.654
86 100.000 100.000 96.938 91.573 84.891
87 100.000 100.000 97.000 91.721 85.126
88 100.000 100.000 97.062 91.866 85.351
89 100.000 100.000 97.124 92.011 85.576
90 100.000 100.000 97.186 92.156 85.801
91 100.000 100.000 97.248 92.300 86.026
92 100.000 100.000 97.310 92.445 86.251
93 100.000 100.000 97.372 92.590 86.476
94 100.000 100.000 97.434 92.735 86.701
95 100.000 100.000 97.496 92.879 86.926
96 100.000 100.000 97.558 93.024 87.151
97 100.000 100.000 97.621 93.160 87.352
98 100.000 100.000 97.683 93.297 87.554
99 100.000 100.000 97.745 93.433 87.756
100 100.000 100.000 97.807 93.570 87.958
101 100.000 100.000 97.869 93.706 88.160
102 100.000 100.000 97.931 93.843 88.362
103 100.000 100.000 97.993 93.979 88.563
104 100.000 100.000 98.055 94.116 88.765
105 100.000 100.000 98.117 94.252 88.967
106 100.000 100.000 98.179 94.389 89.169
107 100.000 100.000 98.241 94.525 89.371
108 100.000 100.000 98.303 94.662 89.573
109 100.000 100.000 98.365 94.798 89.774
110 100.000 100.000 98.427 94.934 89.976
111 100.000 100.000 98.490 95.071 90.178
112 100.000 100.000 98.552 95.207 90.380
113 100.000 100.000 98.614 95.344 90.582
114 100.000 100.000 98.676 95.480 90.784
115 100.000 100.000 98.738 95.617 90.985
116 100.000 100.000 98.800 95.753 91.187
117 100.000 100.000 98.841 95.865 91.378
118 100.000 100.000 98.883 95.977 91.572
119 100.000 100.000 98.924 96.080 91.742
120 100.000 100.000 98.965 96.183 91.912
121 100.000 100.000 98.986 96.254 92.048
122 100.000 100.000 99.007 96.316 92.164
123 100.000 100.000 99.028 96.378 92.279
124 100.000 100.000 99.048 96.440 92.395
125 100.000 100.000 99.069 96.502 92.510
126 100.000 100.000 99.090 96.564 92.626
127 100.000 100.000 99.110 96.626 92.741
128 100.000 100.000 99.131 96.688 92.857
129 100.000 100.000 99.152 96.750 92.972
130 100.000 100.000 99.172 96.812 93.088
131 100.000 100.000 99.193 96.874 93.203
132 100.000 100.000 99.214 96.936 93.319
133 100.000 100.000 99.234 96.998 93.434
134 100.000 100.000 99.255 97.060 93.550
135 100.000 100.000 99.276 97.118 93.653
136 100.000 100.000 99.297 97.176 93.756
137 100.000 100.000 99.317 97.234 93.861
138 100.000 100.000 99.338 97.291 93.964
139 100.000 100.000 99.359 97.349 94.068
140 100.000 100.000 99.379 97.407 94.171
141 100.000 100.000 99.400 97.465 94.274
142 100.000 100.000 99.421 97.523 94.378
143 100.000 100.000 99.441 97.581 94.481
144 100.000 100.000 99.462 97.635 94.574
145 100.000 100.000 99.483 97.688 94.666
146 100.000 100.000 99.503 97.742 94.758
147 100.000 100.000 99.524 97.796 94.851
148 100.000 100.000 99.545 97.850 94.943
149 100.000 100.000 99.565 97.903 95.036
150 100.000 100.000 99.586 97.957 95.128
151 100.000 100.000 99.607 98.011 95.220
152 100.000 100.000 99.628 98.065 95.313
153 100.000 100.000 99.648 98.119 95.405
154 100.000 100.000 99.669 98.172 95.498
155 100.000 100.000 99.690 98.226 95.590
156 100.000 100.000 99.710 98.280 95.682
157 100.000 100.000 99.731 98.334 95.775
158 100.000 100.000 99.752 98.387 95.867
159 100.000 100.000 99.772 98.441 95.960
160 100.000 100.000 99.793 98.495 96.052
161 100.000 100.000 99.814 98.549 96.144
162 100.000 100.000 99.834 98.602 96.237
163 100.000 100.000 99.855 98.656 96.329
164 100.000 100.000 99.876 98.710 96.422
165 100.000 100.000 99.897 98.764 96.514
166 100.000 100.000 99.917 98.817 96.606
167 100.000 100.000 99.938 98.871 96.699
168 100.000 100.000 99.959 98.925 96.791
169 100.000 100.000 99.979 98.979 96.884
170 100.000 100.000 100.000 99.032 96.976
171 100.000 100.000 100.000 99.057 97.045
172 100.000 100.000 100.000 99.074 97.094
173 100.000 100.000 100.000 99.090 97.143
174 100.000 100.000 100.000 99.107 97.191
175 100.000 100.000 100.000 99.119 97.229
176 100.000 100.000 100.000 99.127 97.255
177 100.000 100.000 100.000 99.136 97.280
178 100.000 100.000 100.000 99.144 97.306
179 100.000 100.000 100.000 99.152 97.331
180 100.000 100.000 100.000 99.161 97.357
181 100.000 100.000 100.000 99.169 97.382
182 100.000 100.000 100.000 99.177 97.408
183 100.000 100.000 100.000 99.185 97.433
184 100.000 100.000 100.000 99.194 97.459
185 100.000 100.000 100.000 99.202 97.484
186 100.000 100.000 100.000 99.210 97.510
187 100.000 100.000 100.000 99.218 97.535
188 100.000 100.000 100.000 99.227 97.561
189 100.000 100.000 100.000 99.235 97.586
190 100.000 100.000 100.000 99.243 97.612
191 100.000 100.000 100.000 99.252 97.638
192 100.000 100.000 100.000 99.260 97.663
193 100.000 100.000 100.000 99.268 97.689
194 100.000 100.000 100.000 99.276 97.714
195 100.000 100.000 100.000 99.285 97.740
196 100.000 100.000 100.000 99.293 97.765
197 100.000 100.000 100.000 99.301 97.791
198 100.000 100.000 100.000 99.309 97.816
199 100.000 100.000 100.000 99.318 97.842
200 100.000 100.000 100.000 99.326 97.867
201 100.000 100.000 100.000 99.334 97.893
202 100.000 100.000 100.000 99.343 97.918
203 100.000 100.000 100.000 99.351 97.944
204 100.000 100.000 100.000 99.359 97.969
205 100.000 100.000 100.000 99.367 97.995
206 100.000 100.000 100.000 99.376 98.021
207 100.000 100.000 100.000 99.384 98.046
208 100.000 100.000 100.000 99.392 98.072
209 100.000 100.000 100.000 99.400 98.097
210 100.000 100.000 100.000 99.409 98.123
211 100.000 100.000 100.000 99.417 98.148
212 100.000 100.000 100.000 99.425 98.174
213 100.000 100.000 100.000 99.433 98.199
214 100.000 100.000 100.000 99.442 98.225
215 100.000 100.000 100.000 99.450 98.250
216 100.000 100.000 100.000 99.458 98.276
217 100.000 100.000 100.000 99.467 98.301
218 100.000 100.000 100.000 99.475 98.327
219 100.000 100.000 100.000 99.483 98.352
220 100.000 100.000 100.000 99.491 98.378
221 100.000 100.000 100.000 99.500 98.404
222 100.000 100.000 100.000 99.508 98.429
223 100.000 100.000 100.000 99.516 98.455
224 100.000 100.000 100.000 99.524 98.480
225 100.000 100.000 100.000 99.533 98.506
226 100.000 100.000 100.000 99.541 98.531
227 100.000 100.000 100.000 99.549 98.557
228 100.000 100.000 100.000 99.558 98.582
229 100.000 100.000 100.000 99.566 98.608
230 100.000 100.000 100.000 99.574 98.633
231 100.000 100.000 100.000 99.582 98.659
232 100.000 100.000 100.000 99.591 98.684
233 100.000 100.000 100.000 99.599 98.710
234 100.000 100.000 100.000 99.607 98.735
235 100.000 100.000 100.000 99.615 98.761
236 100.000 100.000 100.000 99.624 98.787
237 100.000 100.000 100.000 99.632 98.812
238 100.000 100.000 100.000 99.640 98.838
239 100.000 100.000 100.000 99.649 98.863
240 100.000 100.000 100.000 99.653 98.877
241 100.000 100.000 100.000 99.657 98.890
242 100.000 100.000 100.000 99.661 98.903
243 100.000 100.000 100.000 99.665 98.917
244 100.000 100.000 100.000 99.669 98.930
245 100.000 100.000 100.000 99.673 98.943
246 100.000 100.000 100.000 99.677 98.957
247 100.000 100.000 100.000 99.682 98.970
248 100.000 100.000 100.000 99.686 98.984
249 100.000 100.000 100.000 99.690 98.997
250 100.000 100.000 100.000 99.694 99.010
251 100.000 100.000 100.000 99.698 99.024
252 100.000 100.000 100.000 99.702 99.037
253 100.000 100.000 100.000 99.706 99.050
254 100.000 100.000 100.000 99.711 99.064
255 100.000 100.000 100.000 99.715 99.077
256 100.000 100.000 100.000 99.719 99.091
257 100.000 100.000 100.000 99.723 99.104
258 100.000 100.000 100.000 99.727 99.117
259 100.000 100.000 100.000 99.731 99.131
260 100.000 100.000 100.000 99.735 99.144
261 100.000 100.000 100.000 99.739 99.157
262 100.000 100.000 100.000 99.744 99.171
263 100.000 100.000 100.000 99.748 99.184
264 100.000 100.000 100.000 99.752 99.198
265 100.000 100.000 100.000 99.756 99.211
266 100.000 100.000 100.000 99.760 99.224
267 100.000 100.000 100.000 99.764 99.238
268 100.000 100.000 100.000 99.768 99.251
269 100.000 100.000 100.000 99.773 99.264
270 100.000 100.000 100.000 99.777 99.278
271 100.000 100.000 100.000 99.781 99.291
272 100.000 100.000 100.000 99.785 99.305
273 100.000 100.000 100.000 99.789 99.318
274 100.000 100.000 100.000 99.793 99.331
275 100.000 100.000 100.000 99.797 99.345
276 100.000 100.000 100.000 99.802 99.358
277 100.000 100.000 100.000 99.806 99.371
278 100.000 100.000 100.000 99.810 99.385
279 100.000 100.000 100.000 99.814 99.398
280 100.000 100.000 100.000 99.818 99.412
281 100.000 100.000 100.000 99.822 99.425
282 100.000 100.000 100.000 99.826 99.438
283 100.000 100.000 100.000 99.830 99.452
284 100.000 100.000 100.000 99.835 99.465
285 100.000 100.000 100.000 99.839 99.478
286 100.000 100.000 100.000 99.843 99.492
287 100.000 100.000 100.000 99.847 99.505
288 100.000 100.000 100.000 99.851 99.519
289 100.000 100.000 100.000 99.855 99.532
290 100.000 100.000 100.000 99.859 99.545
291 100.000 100.000 100.000 99.864 99.559
292 100.000 100.000 100.000 99.868 99.572
293 100.000 100.000 100.000 99.872 99.585
294 100.000 100.000 100.000 99.876 99.599
295 100.000 100.000 100.000 99.880 99.612
296 100.000 100.000 100.000 99.884 99.626
297 100.000 100.000 100.000 99.888 99.639
298 100.000 100.000 100.000 99.892 99.652
299 100.000 100.000 100.000 99.897 99.666
300 100.000 100.000 100.000 99.901 99.679
301 100.000 100.000 100.000 99.905 99.692
302 100.000 100.000 100.000 99.909 99.706
303 100.000 100.000 100.000 99.913 99.719
304 100.000 100.000 100.000 99.917 99.733
305 100.000 100.000 100.000 99.921 99.746
306 100.000 100.000 100.000 99.926 99.759
307 100.000 100.000 100.000 99.930 99.773
308 100.000 100.000 100.000 99.934 99.786
309 100.000 100.000 100.000 99.938 99.799
310 100.000 100.000 100.000 99.942 99.813
311 100.000 100.000 100.000 99.946 99.826
312 100.000 100.000 100.000 99.950 99.840
313 100.000 100.000 100.000 99.955 99.853
314 100.000 100.000 100.000 99.959 99.866
315 100.000 100.000 100.000 99.963 99.880
316 100.000 100.000 100.000 99.967 99.893
317 100.000 100.000 100.000 99.971 99.906
318 100.000 100.000 100.000 99.975 99.920
319 100.000 100.000 100.000 99.979 99.933
320 100.000 100.000 100.000 99.983 99.947
321 100.000 100.000 100.000 99.988 99.960
322 100.000 100.000 100.000 99.992 99.973
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 4, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 55.000 30.233 17.670 10.999 7.234
3 72.500 43.355 26.154 16.458 10.854
4 72.500 48.505 31.616 20.841 14.108
5 77.500 54.319 36.685 24.852 17.160
6 82.500 59.136 40.865 28.301 19.908
7 85.000 62.957 44.672 31.592 22.581
8 92.500 69.601 49.741 35.376 25.416
9 95.000 72.591 52.597 37.894 27.551
10 97.500 75.581 55.431 40.384 29.660
11 100.000 78.073 57.770 42.522 31.557
12 100.000 79.734 59.983 44.709 33.504
13 100.000 80.565 61.535 46.529 35.279
14 100.000 81.728 63.232 48.336 36.987
15 100.000 83.056 65.073 50.225 38.729
16 100.000 84.385 66.894 52.070 40.407
17 100.000 85.382 68.363 53.653 41.926
18 100.000 86.213 69.625 55.055 43.308
19 100.000 87.209 71.053 56.565 44.738
20 100.000 87.375 71.839 57.735 46.022
21 100.000 88.372 73.226 59.174 47.374
22 100.000 89.203 74.447 60.497 48.658
23 100.000 89.535 75.150 61.440 49.704
24 100.000 90.199 76.185 62.606 50.865
25 100.000 90.698 77.012 63.603 51.922
26 100.000 91.196 77.861 64.624 52.992
27 100.000 91.860 78.854 65.720 54.075
28 100.000 92.525 79.847 66.816 55.158
29 100.000 93.189 80.840 67.911 56.242
30 100.000 93.355 81.316 68.627 57.087
31 100.000 93.522 81.792 69.342 57.932
32 100.000 94.352 82.930 70.500 58.996
33 100.000 94.684 83.551 71.290 59.854
34 100.000 94.850 83.985 71.935 60.621
35 100.000 95.183 84.585 72.687 61.436
36 100.000 95.515 85.185 73.440 62.251
37 100.000 95.847 85.765 74.164 63.042
38 100.000 96.013 86.178 74.767 63.755
39 100.000 96.179 86.613 75.396 64.478
40 100.000 96.346 87.006 75.962 65.147
41 100.000 96.678 87.544 76.612 65.850
42 100.000 96.844 87.937 77.174 66.507
43 100.000 97.010 88.310 77.707 67.142
44 100.000 97.176 88.682 78.241 67.777
45 100.000 97.342 89.075 78.799 68.422
46 100.000 97.508 89.468 79.357 69.068
47 100.000 97.674 89.861 79.916 69.714
48 100.000 97.841 90.255 80.474 70.359
49 100.000 98.007 90.627 80.999 70.971
50 100.000 98.173 90.958 81.462 71.528
51 100.000 98.505 91.475 82.066 72.164
52 100.000 98.671 91.806 82.521 72.697
53 100.000 98.837 92.137 82.976 73.231
54 100.000 99.003 92.468 83.431 73.765
55 100.000 99.003 92.613 83.741 74.210
56 100.000 99.003 92.820 84.133 74.715
57 100.000 99.003 93.027 84.526 75.219
58 100.000 99.003 93.234 84.919 75.724
59 100.000 99.169 93.565 85.366 76.237
60 100.000 99.336 93.896 85.812 76.750
61 100.000 99.502 94.227 86.259 77.263
62 100.000 99.668 94.558 86.706 77.776
63 100.000 99.668 94.682 86.983 78.187
64 100.000 99.834 94.993 87.392 78.656
65 100.000 100.000 95.303 87.801 79.126
66 100.000 100.000 95.427 88.062 79.493
67 100.000 100.000 95.551 88.322 79.860
68 100.000 100.000 95.655 88.546 80.184
69 100.000 100.000 95.758 88.769 80.507
70 100.000 100.000 95.841 88.959 80.799
71 100.000 100.000 95.924 89.145 81.080
72 100.000 100.000 96.007 89.327 81.348
73 100.000 100.000 96.089 89.509 81.617
74 100.000 100.000 96.172 89.691 81.886
75 100.000 100.000 96.234 89.844 82.131
76 100.000 100.000 96.296 89.997 82.377
77 100.000 100.000 96.358 90.150 82.623
78 100.000 100.000 96.420 90.303 82.868
79 100.000 100.000 96.483 90.456 83.114
80 100.000 100.000 96.545 90.609 83.359
81 100.000 100.000 96.607 90.762 83.605
82 100.000 100.000 96.669 90.915 83.851
83 100.000 100.000 96.731 91.068 84.096
84 100.000 100.000 96.793 91.221 84.342
85 100.000 100.000 96.855 91.374 84.588
86 100.000 100.000 96.917 91.523 84.822
87 100.000 100.000 96.979 91.672 85.057
88 100.000 100.000 97.041 91.817 85.282
89 100.000 100.000 97.103 91.961 85.507
90 100.000 100.000 97.165 92.106 85.732
91 100.000 100.000 97.227 92.251 85.957
92 100.000 100.000 97.289 92.395 86.182
93 100.000 100.000 97.331 92.507 86.375
94 100.000 100.000 97.372 92.619 86.566
95 100.000 100.000 97.414 92.730 86.757
96 100.000 100.000 97.455 92.842 86.948
97 100.000 100.000 97.496 92.954 87.138
98 100.000 100.000 97.538 93.065 87.329
99 100.000 100.000 97.579 93.177 87.520
100 100.000 100.000 97.641 93.313 87.722
101 100.000 100.000 97.703 93.450 87.924
102 100.000 100.000 97.765 93.586 88.126
103 100.000 100.000 97.827 93.723 88.328
104 100.000 100.000 97.890 93.859 88.529
105 100.000 100.000 97.952 93.996 88.731
106 100.000 100.000 98.014 94.132 88.933
107 100.000 100.000 98.076 94.269 89.135
108 100.000 100.000 98.138 94.405 89.337
109 100.000 100.000 98.200 94.542 89.539
110 100.000 100.000 98.262 94.678 89.740
111 100.000 100.000 98.324 94.815 89.942
112 100.000 100.000 98.386 94.951 90.144
113 100.000 100.000 98.448 95.087 90.346
114 100.000 100.000 98.510 95.224 90.548
115 100.000 100.000 98.572 95.360 90.750
116 100.000 100.000 98.634 95.497 90.951
117 100.000 100.000 98.696 95.633 91.153
118 100.000 100.000 98.759 95.770 91.355
119 100.000 100.000 98.800 95.873 91.525
120 100.000 100.000 98.841 95.977 91.696
121 100.000 100.000 98.883 96.080 91.866
122 100.000 100.000 98.924 96.183 92.036
123 100.000 100.000 98.965 96.287 92.206
124 100.000 100.000 99.007 96.382 92.353
125 100.000 100.000 99.048 96.477 92.500
126 100.000 100.000 99.090 96.572 92.648
127 100.000 100.000 99.110 96.634 92.763
128 100.000 100.000 99.131 96.696 92.879
129 100.000 100.000 99.152 96.758 92.994
130 100.000 100.000 99.172 96.820 93.110
131 100.000 100.000 99.193 96.882 93.225
132 100.000 100.000 99.214 96.944 93.341
133 100.000 100.000 99.234 97.006 93.456
134 100.000 100.000 99.255 97.068 93.572
135 100.000 100.000 99.276 97.130 93.687
136 100.000 100.000 99.297 97.192 93.803
137 100.000 100.000 99.317 97.254 93.918
138 100.000 100.000 99.338 97.316 94.034
139 100.000 100.000 99.359 97.378 94.149
140 100.000 100.000 99.379 97.436 94.254
141 100.000 100.000 99.400 97.494 94.357
142 100.000 100.000 99.421 97.552 94.460
143 100.000 100.000 99.441 97.610 94.564
144 100.000 100.000 99.462 97.668 94.667
145 100.000 100.000 99.483 97.722 94.760
146 100.000 100.000 99.503 97.775 94.852
147 100.000 100.000 99.524 97.829 94.944
148 100.000 100.000 99.545 97.883 95.037
149 100.000 100.000 99.565 97.937 95.129
150 100.000 100.000 99.586 97.990 95.222
151 100.000 100.000 99.607 98.044 95.314
152 100.000 100.000 99.628 98.098 95.406
153 100.000 100.000 99.648 98.152 95.499
154 100.000 100.000 99.669 98.205 95.591
155 100.000 100.000 99.690 98.259 95.684
156 100.000 100.000 99.710 98.313 95.776
157 100.000 100.000 99.731 98.367 95.868
158 100.000 100.000 99.752 98.420 95.961
159 100.000 100.000 99.772 98.474 96.053
160 100.000 100.000 99.793 98.528 96.146
161 100.000 100.000 99.814 98.582 96.238
162 100.000 100.000 99.834 98.635 96.330
163 100.000 100.000 99.855 98.689 96.423
164 100.000 100.000 99.876 98.743 96.515
165 100.000 100.000 99.897 98.797 96.608
166 100.000 100.000 99.917 98.850 96.700
167 100.000 100.000 99.938 98.904 96.793
168 100.000 100.000 99.959 98.958 96.885
169 100.000 100.000 99.979 99.012 96.977
170 100.000 100.000 100.000 99.065 97.070
171 100.000 100.000 100.000 99.082 97.118
172 100.000 100.000 100.000 99.099 97.167
173 100.000 100.000 100.000 99.115 97.216
174 100.000 100.000 100.000 99.132 97.264
175 100.000 100.000 100.000 99.140 97.290
176 100.000 100.000 100.000 99.148 97.315
177 100.000 100.000 100.000 99.156 97.341
178 100.000 100.000 100.000 99.165 97.366
179 100.000 100.000 100.000 99.173 97.392
180 100.000 100.000 100.000 99.181 97.417
181 100.000 100.000 100.000 99.190 97.443
182 100.000 100.000 100.000 99.198 97.469
183 100.000 100.000 100.000 99.206 97.494
184 100.000 100.000 100.000 99.214 97.520
185 100.000 100.000 100.000 99.223 97.545
186 100.000 100.000 100.000 99.231 97.571
187 100.000 100.000 100.000 99.239 97.596
188 100.000 100.000 100.000 99.247 97.622
189 100.000 100.000 100.000 99.256 97.647
190 100.000 100.000 100.000 99.264 97.673
191 100.000 100.000 100.000 99.272 97.698
192 100.000 100.000 100.000 99.280 97.724
193 100.000 100.000 100.000 99.289 97.749
194 100.000 100.000 100.000 99.297 97.775
195 100.000 100.000 100.000 99.305 97.800
196 100.000 100.000 100.000 99.314 97.826
197 100.000 100.000 100.000 99.322 97.852
198 100.000 100.000 100.000 99.330 97.877
199 100.000 100.000 100.000 99.338 97.903
200 100.000 100.000 100.000 99.347 97.928
201 100.000 100.000 100.000 99.355 97.954
202 100.000 100.000 100.000 99.363 97.979
203 100.000 100.000 100.000 99.371 98.005
204 100.000 100.000 100.000 99.380 98.030
205 100.000 100.000 100.000 99.388 98.056
206 100.000 100.000 100.000 99.396 98.081
207 100.000 100.000 100.000 99.405 98.107
208 100.000 100.000 100.000 99.413 98.132
209 100.000 100.000 100.000 99.421 98.158
210 100.000 100.000 100.000 99.429 98.183
211 100.000 100.000 100.000 99.438 98.209
212 100.000 100.000 100.000 99.446 98.235
213 100.000 100.000 100.000 99.454 98.260
214 100.000 100.000 100.000 99.462 98.286
215 100.000 100.000 100.000 99.471 98.311
216 100.000 100.000 100.000 99.479 98.337
217 100.000 100.000 100.000 99.487 98.362
218 100.000 100.000 100.000 99.496 98.388
219 100.000 100.000 100.000 99.504 98.413
220 100.000 100.000 100.000 99.512 98.439
221 100.000 100.000 100.000 99.520 98.464
222 100.000 100.000 100.000 99.529 98.490
223 100.000 100.000 100.000 99.537 98.515
224 100.000 100.000 100.000 99.545 98.541
225 100.000 100.000 100.000 99.553 98.566
226 100.000 100.000 100.000 99.562 98.592
227 100.000 100.000 100.000 99.570 98.618
228 100.000 100.000 100.000 99.578 98.643
229 100.000 100.000 100.000 99.586 98.669
230 100.000 100.000 100.000 99.595 98.694
231 100.000 100.000 100.000 99.603 98.720
232 100.000 100.000 100.000 99.611 98.745
233 100.000 100.000 100.000 99.620 98.771
234 100.000 100.000 100.000 99.628 98.796
235 100.000 100.000 100.000 99.632 98.810
236 100.000 100.000 100.000 99.636 98.823
237 100.000 100.000 100.000 99.640 98.836
238 100.000 100.000 100.000 99.644 98.850
239 100.000 100.000 100.000 99.649 98.863
240 100.000 100.000 100.000 99.653 98.877
241 100.000 100.000 100.000 99.657 98.890
242 100.000 100.000 100.000 99.661 98.903
243 100.000 100.000 100.000 99.665 98.917
244 100.000 100.000 100.000 99.669 98.930
245 100.000 100.000 100.000 99.673 98.943
246 100.000 100.000 100.000 99.677 98.957
247 100.000 100.000 100.000 99.682 98.970
248 100.000 100.000 100.000 99.686 98.984
249 100.000 100.000 100.000 99.690 98.997
250 100.000 100.000 100.000 99.694 99.010
251 100.000 100.000 100.000 99.698 99.024
252 100.000 100.000 100.000 99.702 99.037
253 100.000 100.000 100.000 99.706 99.050
254 100.000 100.000 100.000 99.711 99.064
255 100.000 100.000 100.000 99.715 99.077
256 100.000 100.000 100.000 99.719 99.091
257 100.000 100.000 100.000 99.723 99.104
258 100.000 100.000 100.000 99.727 99.117
259 100.000 100.000 100.000 99.731 99.131
260 100.000 100.000 100.000 99.735 99.144
261 100.000 100.000 100.000 99.739 99.157
262 100.000 100.000 100.000 99.744 99.171
263 100.000 100.000 100.000 99.748 99.184
264 100.000 100.000 100.000 99.752 99.198
265 100.000 100.000 100.000 99.756 99.211
266 100.000 100.000 100.000 99.760 99.224
267 100.000 100.000 100.000 99.764 99.238
268 100.000 100.000 100.000 99.768 99.251
269 100.000 100.000 100.000 99.773 99.264
270 100.000 100.000 100.000 99.777 99.278
271 100.000 100.000 100.000 99.781 99.291
272 100.000 100.000 100.000 99.785 99.305
273 100.000 100.000 100.000 99.789 99.318
274 100.000 100.000 100.000 99.793 99.331
275 100.000 100.000 100.000 99.797 99.345
276 100.000 100.000 100.000 99.802 99.358
277 100.000 100.000 100.000 99.806 99.371
278 100.000 100.000 100.000 99.810 99.385
279 100.000 100.000 100.000 99.814 99.398
280 100.000 100.000 100.000 99.818 99.412
281 100.000 100.000 100.000 99.822 99.425
282 100.000 100.000 100.000 99.826 99.438
283 100.000 100.000 100.000 99.830 99.452
284 100.000 100.000 100.000 99.835 99.465
285 100.000 100.000 100.000 99.839 99.478
286 100.000 100.000 100.000 99.843 99.492
287 100.000 100.000 100.000 99.847 99.505
288 100.000 100.000 100.000 99.851 99.519
289 100.000 100.000 100.000 99.855 99.532
290 100.000 100.000 100.000 99.859 99.545
291 100.000 100.000 100.000 99.864 99.559
292 100.000 100.000 100.000 99.868 99.572
293 100.000 100.000 100.000 99.872 99.585
294 100.000 100.000 100.000 99.876 99.599
295 100.000 100.000 100.000 99.880 99.612
296 100.000 100.000 100.000 99.884 99.626
297 100.000 100.000 100.000 99.888 99.639
298 100.000 100.000 100.000 99.892 99.652
299 100.000 100.000 100.000 99.897 99.666
300 100.000 100.000 100.000 99.901 99.679
301 100.000 100.000 100.000 99.905 99.692
302 100.000 100.000 100.000 99.909 99.706
303 100.000 100.000 100.000 99.913 99.719
304 100.000 100.000 100.000 99.917 99.733
305 100.000 100.000 100.000 99.921 99.746
306 100.000 100.000 100.000 99.926 99.759
307 100.000 100.000 100.000 99.930 99.773
308 100.000 100.000 100.000 99.934 99.786
309 100.000 100.000 100.000 99.938 99.799
310 100.000 100.000 100.000 99.942 99.813
311 100.000 100.000 100.000 99.946 99.826
312 100.000 100.000 100.000 99.950 99.840
313 100.000 100.000 100.000 99.955 99.853
314 100.000 100.000 100.000 99.959 99.866
315 100.000 100.000 100.000 99.963 99.880
316 100.000 100.000 100.000 99.967 99.893
317 100.000 100.000 100.000 99.971 99.906
318 100.000 100.000 100.000 99.975 99.920
319 100.000 100.000 100.000 99.979 99.933
320 100.000 100.000 100.000 99.983 99.947
321 100.000 100.000 100.000 99.988 99.960
322 100.000 100.000 100.000 99.992 99.973
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 5, Prioritisation Strength: 2

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 77.500 49.668 31.616 20.688 14.011
5 80.000 55.648 37.285 25.113 17.270
6 85.000 60.797 41.610 28.549 19.951
7 90.000 65.615 45.541 31.650 22.384
8 92.500 69.269 49.038 34.607 24.775
9 95.000 72.757 52.328 37.377 27.019
10 97.500 75.581 55.224 40.065 29.342
11 100.000 78.405 57.811 42.298 31.227
12 100.000 79.900 59.797 44.258 32.973
13 100.000 81.395 61.887 46.400 34.930
14 100.000 82.558 63.480 48.025 36.428
15 100.000 83.721 65.239 49.932 38.239
16 100.000 84.884 66.812 51.520 39.694
17 100.000 85.880 68.115 52.793 40.826
18 100.000 86.877 69.667 54.489 42.437
19 100.000 87.874 71.053 55.919 43.768
20 100.000 88.870 72.522 57.491 45.257
21 100.000 89.701 74.074 59.310 47.033
22 100.000 90.532 75.253 60.522 48.140
23 100.000 91.196 76.350 61.804 49.444
24 100.000 91.860 77.343 62.900 50.527
25 100.000 92.359 78.212 63.958 51.641
26 100.000 92.857 78.999 64.851 52.535
27 100.000 93.355 79.806 65.802 53.527
28 100.000 93.688 80.447 66.638 54.451
29 100.000 94.020 81.026 67.345 55.198
30 100.000 94.352 81.626 68.097 56.013
31 100.000 94.684 82.226 68.850 56.828
32 100.000 95.017 82.744 69.445 57.443
33 100.000 95.349 83.344 70.190 58.235
34 100.000 95.681 83.902 70.880 58.992
35 100.000 95.847 84.213 71.290 59.461
36 100.000 96.013 84.482 71.612 59.809
37 100.000 96.179 84.833 72.104 60.389
38 100.000 96.346 85.102 72.427 60.737
39 100.000 96.512 85.371 72.749 61.085
40 100.000 96.678 85.661 73.126 61.520
41 100.000 96.844 85.930 73.448 61.868
42 100.000 97.010 86.344 74.044 62.557
43 100.000 97.176 86.613 74.366 62.905
44 100.000 97.342 86.965 74.854 63.473
45 100.000 97.508 87.296 75.301 63.986
46 100.000 97.674 87.565 75.623 64.333
47 100.000 97.841 87.916 76.120 64.924
48 100.000 98.007 88.268 76.599 65.471
49 100.000 98.173 88.599 77.054 66.008
50 100.000 98.339 88.972 77.579 66.619
51 100.000 98.505 89.323 78.059 67.166
52 100.000 98.671 89.737 78.654 67.856
53 100.000 98.837 90.048 79.064 68.325
54 100.000 99.003 90.441 79.622 68.971
55 100.000 99.169 90.792 80.114 69.551
56 100.000 99.336 91.082 80.490 69.986
57 100.000 99.502 91.351 80.813 70.334
58 100.000 99.668 91.703 81.301 70.902
59 100.000 99.834 92.034 81.756 71.435
60 100.000 100.000 92.303 82.078 71.783
61 100.000 100.000 92.406 82.297 72.096
62 100.000 100.000 92.489 82.488 72.385
63 100.000 100.000 92.634 82.806 72.851
64 100.000 100.000 92.655 82.864 72.954
65 100.000 100.000 92.841 83.236 73.454
66 100.000 100.000 92.903 83.373 73.656
67 100.000 100.000 92.986 83.555 73.924
68 100.000 100.000 93.027 83.650 74.071
69 100.000 100.000 93.068 83.753 74.242
70 100.000 100.000 93.151 83.935 74.510
71 100.000 100.000 93.255 84.154 74.823
72 100.000 100.000 93.317 84.291 75.025
73 100.000 100.000 93.379 84.427 75.226
74 100.000 100.000 93.462 84.609 75.495
75 100.000 100.000 93.606 84.911 75.917
76 100.000 100.000 93.710 85.130 76.230
77 100.000 100.000 93.855 85.436 76.664
78 100.000 100.000 93.896 85.564 76.889
79 100.000 100.000 94.062 85.903 77.354
80 100.000 100.000 94.186 86.172 77.745
81 100.000 100.000 94.310 86.433 78.112
82 100.000 100.000 94.351 86.528 78.259
83 100.000 100.000 94.434 86.710 78.528
84 100.000 100.000 94.517 86.883 78.773
85 100.000 100.000 94.558 86.978 78.920
86 100.000 100.000 94.600 87.074 79.067
87 100.000 100.000 94.641 87.177 79.238
88 100.000 100.000 94.703 87.313 79.439
89 100.000 100.000 94.807 87.541 79.775
90 100.000 100.000 94.910 87.764 80.098
91 100.000 100.000 94.951 87.876 80.289
92 100.000 100.000 94.993 87.971 80.437
93 100.000 100.000 95.013 88.021 80.517
94 100.000 100.000 95.055 88.116 80.664
95 100.000 100.000 95.096 88.211 80.811
96 100.000 100.000 95.138 88.306 80.958
97 100.000 100.000 95.241 88.529 81.282
98 100.000 100.000 95.303 88.666 81.483
99 100.000 100.000 95.365 88.802 81.685
100 100.000 100.000 95.386 88.860 81.789
101 100.000 100.000 95.407 88.918 81.892
102 100.000 100.000 95.448 89.038 82.106
103 100.000 100.000 95.489 89.133 82.253
104 100.000 100.000 95.551 89.269 82.455
105 100.000 100.000 95.593 89.364 82.602
106 100.000 100.000 95.655 89.501 82.804
107 100.000 100.000 95.676 89.559 82.907
108 100.000 100.000 95.738 89.699 83.121
109 100.000 100.000 95.800 89.836 83.323
110 100.000 100.000 95.841 89.931 83.470
111 100.000 100.000 95.882 90.026 83.617
112 100.000 100.000 95.882 90.038 83.654
113 100.000 100.000 95.945 90.191 83.899
114 100.000 100.000 95.965 90.241 83.980
115 100.000 100.000 96.007 90.336 84.127
116 100.000 100.000 96.048 90.431 84.274
117 100.000 100.000 96.255 90.824 84.778
118 100.000 100.000 96.276 90.874 84.859
119 100.000 100.000 96.296 90.923 84.939
120 100.000 100.000 96.358 91.072 85.174
121 100.000 100.000 96.379 91.130 85.277
122 100.000 100.000 96.420 91.225 85.424
123 100.000 100.000 96.483 91.362 85.626
124 100.000 100.000 96.607 91.622 85.993
125 100.000 100.000 96.648 91.717 86.140
126 100.000 100.000 96.710 91.870 86.386
127 100.000 100.000 96.731 91.924 86.478
128 100.000 100.000 96.731 91.936 86.515
129 100.000 100.000 96.752 91.986 86.595
130 100.000 100.000 96.855 92.209 86.918
131 100.000 100.000 96.855 92.214 86.932
132 100.000 100.000 96.896 92.309 87.079
133 100.000 100.000 96.917 92.358 87.159
134 100.000 100.000 96.958 92.453 87.306
135 100.000 100.000 97.041 92.627 87.552
136 100.000 100.000 97.062 92.677 87.632
137 100.000 100.000 97.062 92.693 87.681
138 100.000 100.000 97.062 92.697 87.694
139 100.000 100.000 97.062 92.706 87.720
140 100.000 100.000 97.083 92.755 87.800
141 100.000 100.000 97.103 92.817 87.915
142 100.000 100.000 97.103 92.826 87.941
143 100.000 100.000 97.103 92.842 87.990
144 100.000 100.000 97.124 92.892 88.070
145 100.000 100.000 97.145 92.941 88.150
146 100.000 100.000 97.207 93.094 88.396
147 100.000 100.000 97.289 93.276 88.664
148 100.000 100.000 97.310 93.334 88.768
149 100.000 100.000 97.310 93.359 88.837
150 100.000 100.000 97.310 93.367 88.863
151 100.000 100.000 97.310 93.376 88.888
152 100.000 100.000 97.331 93.429 88.980
153 100.000 100.000 97.331 93.438 89.006
154 100.000 100.000 97.331 93.450 89.042
155 100.000 100.000 97.455 93.710 89.410
156 100.000 100.000 97.455 93.719 89.435
157 100.000 100.000 97.476 93.772 89.528
158 100.000 100.000 97.496 93.826 89.620
159 100.000 100.000 97.558 93.979 89.866
160 100.000 100.000 97.558 93.983 89.879
161 100.000 100.000 97.579 94.041 89.982
162 100.000 100.000 97.621 94.136 90.129
163 100.000 100.000 97.641 94.186 90.210
164 100.000 100.000 97.683 94.281 90.357
165 100.000 100.000 97.703 94.339 90.460
166 100.000 100.000 97.703 94.343 90.474
167 100.000 100.000 97.703 94.351 90.499
168 100.000 100.000 97.724 94.409 90.602
169 100.000 100.000 97.765 94.504 90.750
170 100.000 100.000 97.765 94.513 90.775
171 100.000 100.000 97.765 94.517 90.788
172 100.000 100.000 97.786 94.575 90.892
173 100.000 100.000 97.786 94.579 90.905
174 100.000 100.000 98.407 95.633 92.121
175 100.000 100.000 98.614 96.026 92.626
176 100.000 100.000 98.614 96.034 92.651
177 100.000 100.000 98.634 96.084 92.731
178 100.000 100.000 98.676 96.179 92.879
179 100.000 100.000 98.676 96.192 92.915
180 100.000 100.000 98.676 96.200 92.941
181 100.000 100.000 98.676 96.208 92.966
182 100.000 100.000 98.676 96.216 92.992
183 100.000 100.000 98.676 96.220 93.005
184 100.000 100.000 98.676 96.245 93.074
185 100.000 100.000 98.717 96.340 93.221
186 100.000 100.000 98.779 96.477 93.423
187 100.000 100.000 98.779 96.493 93.472
188 100.000 100.000 98.800 96.543 93.552
189 100.000 100.000 98.800 96.551 93.578
190 100.000 100.000 98.800 96.576 93.647
191 100.000 100.000 98.821 96.634 93.750
192 100.000 100.000 98.821 96.638 93.764
193 100.000 100.000 98.821 96.646 93.789
194 100.000 100.000 98.821 96.651 93.803
195 100.000 100.000 98.841 96.700 93.883
196 100.000 100.000 98.841 96.708 93.908
197 100.000 100.000 98.903 96.845 94.110
198 100.000 100.000 98.903 96.849 94.124
199 100.000 100.000 98.945 96.944 94.271
200 100.000 100.000 98.965 96.994 94.351
201 100.000 100.000 98.965 97.002 94.377
202 100.000 100.000 98.986 97.052 94.457
203 100.000 100.000 99.028 97.147 94.604
204 100.000 100.000 99.028 97.151 94.617
205 100.000 100.000 99.048 97.201 94.698
206 100.000 100.000 99.048 97.205 94.711
207 100.000 100.000 99.069 97.254 94.791
208 100.000 100.000 99.090 97.308 94.884
209 100.000 100.000 99.110 97.358 94.964
210 100.000 100.000 99.110 97.382 95.033
211 100.000 100.000 99.110 97.399 95.082
212 100.000 100.000 99.131 97.449 95.162
213 100.000 100.000 99.131 97.457 95.188
214 100.000 100.000 99.131 97.461 95.201
215 100.000 100.000 99.131 97.478 95.250
216 100.000 100.000 99.172 97.573 95.397
217 100.000 100.000 99.214 97.668 95.544
218 100.000 100.000 99.214 97.676 95.569
219 100.000 100.000 99.214 97.684 95.595
220 100.000 100.000 99.214 97.693 95.620
221 100.000 100.000 99.255 97.788 95.768
222 100.000 100.000 99.255 97.796 95.793
223 100.000 100.000 99.297 97.891 95.940
224 100.000 100.000 99.338 97.986 96.087
225 100.000 100.000 99.359 98.036 96.168
226 100.000 100.000 99.359 98.044 96.193
227 100.000 100.000 99.359 98.052 96.219
228 100.000 100.000 99.379 98.106 96.311
229 100.000 100.000 99.379 98.114 96.337
230 100.000 100.000 99.379 98.119 96.350
231 100.000 100.000 99.379 98.127 96.375
232 100.000 100.000 99.379 98.143 96.424
233 100.000 100.000 99.379 98.152 96.450
234 100.000 100.000 99.379 98.160 96.475
235 100.000 100.000 99.379 98.168 96.501
236 100.000 100.000 99.379 98.172 96.514
237 100.000 100.000 99.379 98.176 96.527
238 100.000 100.000 99.379 98.181 96.541
239 100.000 100.000 99.400 98.234 96.633
240 100.000 100.000 99.400 98.243 96.659
241 100.000 100.000 99.441 98.338 96.806
242 100.000 100.000 99.462 98.387 96.886
243 100.000 100.000 99.503 98.482 97.033
244 100.000 100.000 99.503 98.491 97.059
245 100.000 100.000 99.503 98.495 97.072
246 100.000 100.000 99.503 98.499 97.086
247 100.000 100.000 99.503 98.515 97.134
248 100.000 100.000 99.524 98.565 97.214
249 100.000 100.000 99.524 98.569 97.228
250 100.000 100.000 99.524 98.573 97.241
251 100.000 100.000 99.524 98.586 97.278
252 100.000 100.000 99.524 98.602 97.326
253 100.000 100.000 99.524 98.611 97.352
254 100.000 100.000 99.545 98.660 97.432
255 100.000 100.000 99.545 98.668 97.458
256 100.000 100.000 99.545 98.677 97.483
257 100.000 100.000 99.545 98.685 97.509
258 100.000 100.000 99.545 98.693 97.534
259 100.000 100.000 99.545 98.697 97.548
260 100.000 100.000 99.545 98.706 97.573
261 100.000 100.000 99.545 98.714 97.599
262 100.000 100.000 99.545 98.726 97.635
263 100.000 100.000 99.545 98.731 97.648
264 100.000 100.000 99.545 98.735 97.662
265 100.000 100.000 99.545 98.747 97.698
266 100.000 100.000 99.545 98.751 97.712
267 100.000 100.000 99.545 98.755 97.725
268 100.000 100.000 99.545 98.759 97.738
269 100.000 100.000 99.545 98.768 97.764
270 100.000 100.000 99.545 98.776 97.790
271 100.000 100.000 99.586 98.871 97.937
272 100.000 100.000 99.586 98.879 97.962
273 100.000 100.000 99.586 98.888 97.988
274 100.000 100.000 99.607 98.937 98.068
275 100.000 100.000 99.607 98.946 98.094
276 100.000 100.000 99.607 98.950 98.107
277 100.000 100.000 99.607 98.958 98.132
278 100.000 100.000 99.628 99.008 98.213
279 100.000 100.000 99.648 99.057 98.293
280 100.000 100.000 99.648 99.065 98.318
281 100.000 100.000 99.648 99.074 98.344
282 100.000 100.000 99.669 99.123 98.424
283 100.000 100.000 99.690 99.173 98.504
284 100.000 100.000 99.731 99.268 98.652
285 100.000 100.000 99.731 99.276 98.677
286 100.000 100.000 99.731 99.280 98.690
287 100.000 100.000 99.731 99.285 98.704
288 100.000 100.000 99.731 99.293 98.729
289 100.000 100.000 99.772 99.388 98.877
290 100.000 100.000 99.772 99.392 98.890
291 100.000 100.000 99.772 99.396 98.903
292 100.000 100.000 99.772 99.400 98.917
293 100.000 100.000 99.772 99.405 98.930
294 100.000 100.000 99.772 99.409 98.943
295 100.000 100.000 99.772 99.413 98.957
296 100.000 100.000 99.772 99.421 98.982
297 100.000 100.000 99.772 99.425 98.996
298 100.000 100.000 99.772 99.429 99.009
299 100.000 100.000 99.793 99.479 99.089
300 100.000 100.000 99.793 99.483 99.103
301 100.000 100.000 99.793 99.487 99.116
302 100.000 100.000 99.793 99.491 99.129
303 100.000 100.000 99.793 99.496 99.143
304 100.000 100.000 99.793 99.504 99.168
305 100.000 100.000 99.793 99.508 99.182
306 100.000 100.000 99.793 99.512 99.195
307 100.000 100.000 99.793 99.516 99.208
308 100.000 100.000 99.793 99.524 99.234
309 100.000 100.000 99.793 99.529 99.247
310 100.000 100.000 99.793 99.533 99.261
311 100.000 100.000 99.793 99.541 99.286
312 100.000 100.000 99.793 99.545 99.300
313 100.000 100.000 99.793 99.553 99.325
314 100.000 100.000 99.793 99.558 99.339
315 100.000 100.000 100.000 99.950 99.843
316 100.000 100.000 100.000 99.955 99.857
317 100.000 100.000 100.000 99.959 99.870
318 100.000 100.000 100.000 99.963 99.883
319 100.000 100.000 100.000 99.967 99.897
320 100.000 100.000 100.000 99.971 99.910
321 100.000 100.000 100.000 99.979 99.936
322 100.000 100.000 100.000 99.983 99.949
323 100.000 100.000 100.000 99.992 99.974
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 5, Prioritisation Strength: 3

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 62.500 33.223 18.622 11.268 7.301
3 72.500 42.857 25.781 16.268 10.775
4 75.000 48.837 31.450 20.692 14.035
5 80.000 55.648 37.285 25.113 17.270
6 85.000 60.797 41.817 28.826 20.188
7 87.500 64.784 45.665 32.064 22.783
8 92.500 69.601 49.597 35.165 25.216
9 95.000 72.757 52.700 37.919 27.531
10 97.500 75.748 55.493 40.338 29.563
11 100.000 78.405 58.018 42.617 31.546
12 100.000 79.734 59.942 44.643 33.434
13 100.000 81.229 61.928 46.603 35.180
14 100.000 82.226 63.646 48.538 37.024
15 100.000 83.555 65.467 50.391 38.722
16 100.000 84.884 67.287 52.243 40.421
17 100.000 85.880 68.860 54.001 42.130
18 100.000 87.043 70.495 55.704 43.727
19 100.000 88.040 71.881 57.143 45.079
20 100.000 89.037 73.267 58.574 46.410
21 100.000 90.033 74.612 59.935 47.664
22 100.000 90.864 75.750 61.093 48.728
23 100.000 91.528 76.805 62.308 49.964
24 100.000 92.027 77.674 63.367 51.078
25 100.000 92.027 78.295 64.421 52.294
26 100.000 92.525 79.123 65.401 53.309
27 100.000 92.857 79.826 66.344 54.355
28 100.000 93.355 80.592 67.213 55.237
29 100.000 93.688 81.233 68.044 56.151
30 100.000 94.020 81.854 68.833 57.009
31 100.000 94.352 82.475 69.623 57.867
32 100.000 94.684 83.095 70.417 58.737
33 100.000 95.017 83.716 71.207 59.595
34 100.000 95.349 84.316 71.976 60.451
35 100.000 95.681 84.916 72.729 61.266
36 100.000 95.847 85.392 73.444 62.111
37 100.000 96.179 85.930 74.081 62.781
38 100.000 96.346 86.365 74.726 63.548
39 100.000 96.512 86.799 75.371 64.315
40 100.000 96.678 87.213 75.975 65.028
41 100.000 96.844 87.627 76.579 65.740
42 100.000 97.010 88.041 77.182 66.453
43 100.000 97.176 88.434 77.741 67.098
44 100.000 97.342 88.827 78.299 67.744
45 100.000 97.508 89.220 78.857 68.390
46 100.000 97.674 89.613 79.415 69.035
47 100.000 97.841 90.006 79.974 69.681
48 100.000 98.007 90.399 80.532 70.326
49 100.000 98.173 90.792 81.090 70.972
50 100.000 98.339 91.144 81.578 71.540
51 100.000 98.505 91.496 82.066 72.108
52 100.000 98.671 91.848 82.558 72.688
53 100.000 98.837 92.179 83.013 73.224
54 100.000 99.003 92.510 83.468 73.758
55 100.000 99.169 92.841 83.927 74.304
56 100.000 99.336 93.172 84.373 74.817
57 100.000 99.502 93.503 84.828 75.350
58 100.000 99.668 93.834 85.283 75.884
59 100.000 99.834 94.165 85.730 76.397
60 100.000 100.000 94.475 86.139 76.867
61 100.000 100.000 94.682 86.532 77.371
62 100.000 100.000 94.889 86.925 77.876
63 100.000 100.000 95.096 87.318 78.380
64 100.000 100.000 95.220 87.586 78.771
65 100.000 100.000 95.345 87.863 79.182
66 100.000 100.000 95.469 88.124 79.549
67 100.000 100.000 95.593 88.384 79.916
68 100.000 100.000 95.696 88.608 80.240
69 100.000 100.000 95.800 88.835 80.573
70 100.000 100.000 95.903 89.058 80.896
71 100.000 100.000 95.986 89.240 81.165
72 100.000 100.000 96.069 89.422 81.434
73 100.000 100.000 96.131 89.559 81.635
74 100.000 100.000 96.193 89.695 81.837
75 100.000 100.000 96.255 89.832 82.039
76 100.000 100.000 96.317 89.968 82.241
77 100.000 100.000 96.379 90.105 82.443
78 100.000 100.000 96.441 90.241 82.645
79 100.000 100.000 96.503 90.378 82.846
80 100.000 100.000 96.565 90.514 83.048
81 100.000 100.000 96.627 90.650 83.250
82 100.000 100.000 96.689 90.795 83.475
83 100.000 100.000 96.752 90.932 83.677
84 100.000 100.000 96.814 91.076 83.902
85 100.000 100.000 96.876 91.213 84.104
86 100.000 100.000 96.938 91.358 84.329
87 100.000 100.000 97.000 91.494 84.530
88 100.000 100.000 97.062 91.647 84.776
89 100.000 100.000 97.124 91.800 85.022
90 100.000 100.000 97.186 91.936 85.223
91 100.000 100.000 97.248 92.073 85.425
92 100.000 100.000 97.310 92.209 85.627
93 100.000 100.000 97.372 92.346 85.829
94 100.000 100.000 97.434 92.499 86.075
95 100.000 100.000 97.496 92.635 86.276
96 100.000 100.000 97.558 92.788 86.522
97 100.000 100.000 97.621 92.941 86.768
98 100.000 100.000 97.683 93.094 87.013
99 100.000 100.000 97.745 93.247 87.259
100 100.000 100.000 97.807 93.400 87.504
101 100.000 100.000 97.869 93.537 87.706
102 100.000 100.000 97.931 93.673 87.908
103 100.000 100.000 97.993 93.826 88.154
104 100.000 100.000 98.055 93.963 88.356
105 100.000 100.000 98.117 94.099 88.557
106 100.000 100.000 98.179 94.236 88.759
107 100.000 100.000 98.241 94.372 88.961
108 100.000 100.000 98.303 94.509 89.163
109 100.000 100.000 98.365 94.662 89.408
110 100.000 100.000 98.427 94.798 89.610
111 100.000 100.000 98.490 94.934 89.812
112 100.000 100.000 98.552 95.071 90.014
113 100.000 100.000 98.614 95.207 90.216
114 100.000 100.000 98.676 95.344 90.418
115 100.000 100.000 98.738 95.480 90.619
116 100.000 100.000 98.800 95.617 90.821
117 100.000 100.000 98.841 95.720 90.992
118 100.000 100.000 98.883 95.824 91.162
119 100.000 100.000 98.924 95.927 91.332
120 100.000 100.000 98.965 96.022 91.479
121 100.000 100.000 98.986 96.076 91.572
122 100.000 100.000 99.007 96.134 91.675
123 100.000 100.000 99.028 96.187 91.767
124 100.000 100.000 99.048 96.258 91.903
125 100.000 100.000 99.069 96.316 92.007
126 100.000 100.000 99.090 96.373 92.110
127 100.000 100.000 99.110 96.427 92.203
128 100.000 100.000 99.131 96.481 92.295
129 100.000 100.000 99.152 96.531 92.375
130 100.000 100.000 99.172 96.584 92.468
131 100.000 100.000 99.193 96.642 92.571
132 100.000 100.000 99.214 96.713 92.707
133 100.000 100.000 99.234 96.775 92.823
134 100.000 100.000 99.255 96.828 92.915
135 100.000 100.000 99.276 96.882 93.007
136 100.000 100.000 99.297 96.932 93.088
137 100.000 100.000 99.317 96.981 93.168
138 100.000 100.000 99.338 97.039 93.271
139 100.000 100.000 99.359 97.089 93.352
140 100.000 100.000 99.379 97.143 93.444
141 100.000 100.000 99.400 97.213 93.580
142 100.000 100.000 99.421 97.271 93.684
143 100.000 100.000 99.441 97.329 93.787
144 100.000 100.000 99.462 97.382 93.879
145 100.000 100.000 99.483 97.436 93.972
146 100.000 100.000 99.503 97.507 94.108
147 100.000 100.000 99.524 97.569 94.223
148 100.000 100.000 99.545 97.622 94.316
149 100.000 100.000 99.565 97.676 94.408
150 100.000 100.000 99.586 97.734 94.512
151 100.000 100.000 99.607 97.796 94.627
152 100.000 100.000 99.628 97.850 94.719
153 100.000 100.000 99.648 97.908 94.823
154 100.000 100.000 99.669 97.961 94.915
155 100.000 100.000 99.690 98.015 95.008
156 100.000 100.000 99.710 98.069 95.100
157 100.000 100.000 99.731 98.123 95.192
158 100.000 100.000 99.752 98.176 95.285
159 100.000 100.000 99.772 98.230 95.377
160 100.000 100.000 99.793 98.288 95.481
161 100.000 100.000 99.814 98.342 95.573
162 100.000 100.000 99.834 98.396 95.665
163 100.000 100.000 99.855 98.453 95.769
164 100.000 100.000 99.876 98.515 95.884
165 100.000 100.000 99.897 98.573 95.988
166 100.000 100.000 99.917 98.623 96.068
167 100.000 100.000 99.938 98.677 96.160
168 100.000 100.000 99.959 98.739 96.276
169 100.000 100.000 99.979 98.793 96.368
170 100.000 100.000 100.000 98.842 96.448
171 100.000 100.000 100.000 98.850 96.474
172 100.000 100.000 100.000 98.855 96.487
173 100.000 100.000 100.000 98.863 96.513
174 100.000 100.000 100.000 98.871 96.538
175 100.000 100.000 100.000 98.875 96.552
176 100.000 100.000 100.000 98.884 96.577
177 100.000 100.000 100.000 98.908 96.647
178 100.000 100.000 100.000 98.917 96.672
179 100.000 100.000 100.000 98.929 96.709
180 100.000 100.000 100.000 98.933 96.722
181 100.000 100.000 100.000 98.946 96.758
182 100.000 100.000 100.000 98.954 96.784
183 100.000 100.000 100.000 98.958 96.797
184 100.000 100.000 100.000 98.966 96.823
185 100.000 100.000 100.000 98.983 96.872
186 100.000 100.000 100.000 98.991 96.897
187 100.000 100.000 100.000 98.995 96.910
188 100.000 100.000 100.000 99.008 96.947
189 100.000 100.000 100.000 99.024 96.996
190 100.000 100.000 100.000 99.032 97.021
191 100.000 100.000 100.000 99.041 97.047
192 100.000 100.000 100.000 99.053 97.083
193 100.000 100.000 100.000 99.057 97.096
194 100.000 100.000 100.000 99.065 97.122
195 100.000 100.000 100.000 99.070 97.135
196 100.000 100.000 100.000 99.082 97.172
197 100.000 100.000 100.000 99.090 97.197
198 100.000 100.000 100.000 99.099 97.223
199 100.000 100.000 100.000 99.107 97.248
200 100.000 100.000 100.000 99.115 97.274
201 100.000 100.000 100.000 99.119 97.287
202 100.000 100.000 100.000 99.127 97.313
203 100.000 100.000 100.000 99.132 97.326
204 100.000 100.000 100.000 99.136 97.340
205 100.000 100.000 100.000 99.140 97.353
206 100.000 100.000 100.000 99.144 97.366
207 100.000 100.000 100.000 99.152 97.392
208 100.000 100.000 100.000 99.161 97.417
209 100.000 100.000 100.000 99.169 97.443
210 100.000 100.000 100.000 99.177 97.469
211 100.000 100.000 100.000 99.181 97.482
212 100.000 100.000 100.000 99.190 97.507
213 100.000 100.000 100.000 99.198 97.533
214 100.000 100.000 100.000 99.214 97.582
215 100.000 100.000 100.000 99.218 97.595
216 100.000 100.000 100.000 99.227 97.621
217 100.000 100.000 100.000 99.235 97.646
218 100.000 100.000 100.000 99.243 97.672
219 100.000 100.000 100.000 99.252 97.697
220 100.000 100.000 100.000 99.256 97.710
221 100.000 100.000 100.000 99.264 97.736
222 100.000 100.000 100.000 99.268 97.749
223 100.000 100.000 100.000 99.276 97.775
224 100.000 100.000 100.000 99.285 97.800
225 100.000 100.000 100.000 99.297 97.837
226 100.000 100.000 100.000 99.305 97.862
227 100.000 100.000 100.000 99.309 97.876
228 100.000 100.000 100.000 99.314 97.889
229 100.000 100.000 100.000 99.338 97.959
230 100.000 100.000 100.000 99.343 97.972
231 100.000 100.000 100.000 99.351 97.997
232 100.000 100.000 100.000 99.355 98.011
233 100.000 100.000 100.000 99.363 98.036
234 100.000 100.000 100.000 99.367 98.050
235 100.000 100.000 100.000 99.376 98.075
236 100.000 100.000 100.000 99.384 98.101
237 100.000 100.000 100.000 99.392 98.126
238 100.000 100.000 100.000 99.400 98.152
239 100.000 100.000 100.000 99.405 98.165
240 100.000 100.000 100.000 99.417 98.202
241 100.000 100.000 100.000 99.421 98.215
242 100.000 100.000 100.000 99.425 98.228
243 100.000 100.000 100.000 99.429 98.242
244 100.000 100.000 100.000 99.442 98.278
245 100.000 100.000 100.000 99.446 98.292
246 100.000 100.000 100.000 99.450 98.305
247 100.000 100.000 100.000 99.475 98.374
248 100.000 100.000 100.000 99.483 98.400
249 100.000 100.000 100.000 99.491 98.425
250 100.000 100.000 100.000 99.496 98.439
251 100.000 100.000 100.000 99.500 98.452
252 100.000 100.000 100.000 99.504 98.466
253 100.000 100.000 100.000 99.512 98.491
254 100.000 100.000 100.000 99.520 98.517
255 100.000 100.000 100.000 99.529 98.542
256 100.000 100.000 100.000 99.537 98.568
257 100.000 100.000 100.000 99.545 98.593
258 100.000 100.000 100.000 99.549 98.607
259 100.000 100.000 100.000 99.558 98.632
260 100.000 100.000 100.000 99.566 98.658
261 100.000 100.000 100.000 99.570 98.671
262 100.000 100.000 100.000 99.578 98.697
263 100.000 100.000 100.000 99.582 98.710
264 100.000 100.000 100.000 99.586 98.723
265 100.000 100.000 100.000 99.595 98.749
266 100.000 100.000 100.000 99.599 98.762
267 100.000 100.000 100.000 99.603 98.776
268 100.000 100.000 100.000 99.611 98.801
269 100.000 100.000 100.000 99.620 98.827
270 100.000 100.000 100.000 99.624 98.840
271 100.000 100.000 100.000 99.628 98.853
272 100.000 100.000 100.000 99.636 98.879
273 100.000 100.000 100.000 99.640 98.892
274 100.000 100.000 100.000 99.644 98.906
275 100.000 100.000 100.000 99.653 98.931
276 100.000 100.000 100.000 99.661 98.957
277 100.000 100.000 100.000 99.686 99.026
278 100.000 100.000 100.000 99.690 99.039
279 100.000 100.000 100.000 99.694 99.053
280 100.000 100.000 100.000 99.698 99.066
281 100.000 100.000 100.000 99.706 99.092
282 100.000 100.000 100.000 99.715 99.117
283 100.000 100.000 100.000 99.723 99.143
284 100.000 100.000 100.000 99.727 99.156
285 100.000 100.000 100.000 99.731 99.170
286 100.000 100.000 100.000 99.739 99.195
287 100.000 100.000 100.000 99.748 99.221
288 100.000 100.000 100.000 99.752 99.234
289 100.000 100.000 100.000 99.768 99.283
290 100.000 100.000 100.000 99.777 99.308
291 100.000 100.000 100.000 99.781 99.322
292 100.000 100.000 100.000 99.785 99.335
293 100.000 100.000 100.000 99.793 99.360
294 100.000 100.000 100.000 99.802 99.386
295 100.000 100.000 100.000 99.806 99.399
296 100.000 100.000 100.000 99.810 99.413
297 100.000 100.000 100.000 99.826 99.461
298 100.000 100.000 100.000 99.830 99.475
299 100.000 100.000 100.000 99.835 99.488
300 100.000 100.000 100.000 99.839 99.501
301 100.000 100.000 100.000 99.843 99.515
302 100.000 100.000 100.000 99.847 99.528
303 100.000 100.000 100.000 99.855 99.554
304 100.000 100.000 100.000 99.864 99.579
305 100.000 100.000 100.000 99.888 99.649
306 100.000 100.000 100.000 99.892 99.662
307 100.000 100.000 100.000 99.901 99.688
308 100.000 100.000 100.000 99.905 99.701
309 100.000 100.000 100.000 99.913 99.726
310 100.000 100.000 100.000 99.921 99.752
311 100.000 100.000 100.000 99.926 99.765
312 100.000 100.000 100.000 99.934 99.791
313 100.000 100.000 100.000 99.938 99.804
314 100.000 100.000 100.000 99.942 99.818
315 100.000 100.000 100.000 99.950 99.843
316 100.000 100.000 100.000 99.959 99.869
317 100.000 100.000 100.000 99.963 99.882
318 100.000 100.000 100.000 99.967 99.895
319 100.000 100.000 100.000 99.975 99.921
320 100.000 100.000 100.000 99.979 99.934
321 100.000 100.000 100.000 99.983 99.948
322 100.000 100.000 100.000 99.992 99.973
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 5, Prioritisation Strength: 4

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 60.000 32.392 18.415 11.227 7.295
3 72.500 43.355 26.154 16.458 10.854
4 75.000 49.668 32.050 20.986 14.150
5 82.500 56.478 37.430 25.080 17.220
6 85.000 61.296 42.127 28.934 20.198
7 90.000 66.113 46.307 32.382 22.946
8 92.500 69.767 49.866 35.438 25.440
9 95.000 72.924 52.866 38.043 27.617
10 97.500 75.914 55.721 40.562 29.753
11 100.000 78.738 58.370 42.902 31.760
12 100.000 80.233 60.439 45.011 33.682
13 100.000 81.229 62.156 46.946 35.526
14 100.000 82.724 64.142 48.906 37.272
15 100.000 83.887 65.880 50.751 38.985
16 100.000 85.050 67.515 52.454 40.581
17 100.000 86.213 69.108 54.083 42.090
18 100.000 87.043 70.412 55.560 43.560
19 100.000 88.040 71.839 57.069 44.990
20 100.000 89.037 73.246 58.541 46.376
21 100.000 89.037 73.867 59.596 47.592
22 100.000 89.701 74.943 60.849 48.872
23 100.000 90.698 76.288 62.209 50.126
24 100.000 91.362 77.281 63.305 51.209
25 100.000 91.694 77.985 64.248 52.255
26 100.000 92.525 79.123 65.405 53.319
27 100.000 93.023 79.950 66.385 54.334
28 100.000 93.522 80.778 67.366 55.349
29 100.000 93.854 81.440 68.238 56.317
30 100.000 94.186 82.082 69.069 57.230
31 100.000 94.518 82.723 69.900 58.143
32 100.000 94.850 83.344 70.694 59.013
33 100.000 95.183 83.964 71.484 59.871
34 100.000 95.515 84.564 72.237 60.686
35 100.000 95.847 85.164 72.981 61.477
36 100.000 96.013 85.599 73.626 62.245
37 100.000 96.179 86.034 74.271 63.012
38 100.000 96.346 86.468 74.916 63.779
39 100.000 96.512 86.903 75.557 64.535
40 100.000 96.678 87.337 76.194 65.282
41 100.000 96.844 87.751 76.802 66.005
42 100.000 97.010 88.144 77.360 66.651
43 100.000 97.176 88.537 77.918 67.296
44 100.000 97.342 88.930 78.477 67.942
45 100.000 97.508 89.323 79.035 68.588
46 100.000 97.674 89.696 79.568 69.222
47 100.000 98.007 90.213 80.172 69.858
48 100.000 98.173 90.565 80.672 70.459
49 100.000 98.339 90.917 81.173 71.060
50 100.000 98.505 91.268 81.665 71.640
51 100.000 98.671 91.620 82.153 72.207
52 100.000 98.837 91.972 82.641 72.775
53 100.000 99.003 92.303 83.096 73.309
54 100.000 99.169 92.634 83.550 73.843
55 100.000 99.336 92.965 84.005 74.377
56 100.000 99.502 93.296 84.460 74.910
57 100.000 99.502 93.503 84.853 75.415
58 100.000 99.668 93.834 85.304 75.939
59 100.000 99.668 94.041 85.697 76.444
60 100.000 99.668 94.248 86.089 76.948
61 100.000 99.834 94.579 86.536 77.461
62 100.000 100.000 94.910 86.983 77.974
63 100.000 100.000 95.055 87.293 78.419
64 100.000 100.000 95.179 87.570 78.830
65 100.000 100.000 95.303 87.830 79.198
66 100.000 100.000 95.427 88.091 79.565
67 100.000 100.000 95.551 88.351 79.932
68 100.000 100.000 95.676 88.612 80.299
69 100.000 100.000 95.779 88.835 80.623
70 100.000 100.000 95.882 89.058 80.946
71 100.000 100.000 95.986 89.282 81.269
72 100.000 100.000 96.069 89.464 81.538
73 100.000 100.000 96.151 89.646 81.807
74 100.000 100.000 96.214 89.799 82.052
75 100.000 100.000 96.276 89.952 82.298
76 100.000 100.000 96.338 90.105 82.544
77 100.000 100.000 96.400 90.258 82.789
78 100.000 100.000 96.462 90.411 83.035
79 100.000 100.000 96.524 90.564 83.280
80 100.000 100.000 96.586 90.717 83.526
81 100.000 100.000 96.648 90.870 83.772
82 100.000 100.000 96.710 91.023 84.017
83 100.000 100.000 96.772 91.176 84.263
84 100.000 100.000 96.834 91.329 84.508
85 100.000 100.000 96.896 91.477 84.746
86 100.000 100.000 96.958 91.622 84.971
87 100.000 100.000 97.020 91.767 85.195
88 100.000 100.000 97.083 91.912 85.420
89 100.000 100.000 97.145 92.056 85.645
90 100.000 100.000 97.207 92.201 85.870
91 100.000 100.000 97.269 92.346 86.095
92 100.000 100.000 97.331 92.491 86.320
93 100.000 100.000 97.393 92.635 86.545
94 100.000 100.000 97.455 92.780 86.770
95 100.000 100.000 97.517 92.925 86.995
96 100.000 100.000 97.579 93.061 87.197
97 100.000 100.000 97.641 93.198 87.399
98 100.000 100.000 97.703 93.334 87.600
99 100.000 100.000 97.765 93.471 87.802
100 100.000 100.000 97.827 93.607 88.004
101 100.000 100.000 97.890 93.744 88.206
102 100.000 100.000 97.952 93.880 88.408
103 100.000 100.000 98.014 94.016 88.610
104 100.000 100.000 98.076 94.153 88.811
105 100.000 100.000 98.138 94.289 89.013
106 100.000 100.000 98.200 94.426 89.215
107 100.000 100.000 98.262 94.562 89.417
108 100.000 100.000 98.324 94.699 89.619
109 100.000 100.000 98.386 94.835 89.821
110 100.000 100.000 98.448 94.972 90.022
111 100.000 100.000 98.510 95.108 90.224
112 100.000 100.000 98.572 95.245 90.426
113 100.000 100.000 98.634 95.381 90.628
114 100.000 100.000 98.696 95.518 90.830
115 100.000 100.000 98.759 95.654 91.032
116 100.000 100.000 98.821 95.790 91.234
117 100.000 100.000 98.862 95.902 91.427
118 100.000 100.000 98.903 96.005 91.597
119 100.000 100.000 98.945 96.109 91.767
120 100.000 100.000 98.965 96.179 91.903
121 100.000 100.000 98.986 96.249 92.040
122 100.000 100.000 99.007 96.311 92.155
123 100.000 100.000 99.028 96.373 92.271
124 100.000 100.000 99.048 96.436 92.386
125 100.000 100.000 99.069 96.498 92.502
126 100.000 100.000 99.090 96.560 92.617
127 100.000 100.000 99.110 96.622 92.733
128 100.000 100.000 99.131 96.684 92.848
129 100.000 100.000 99.152 96.746 92.964
130 100.000 100.000 99.172 96.808 93.079
131 100.000 100.000 99.193 96.870 93.195
132 100.000 100.000 99.214 96.932 93.310
133 100.000 100.000 99.234 96.994 93.426
134 100.000 100.000 99.255 97.056 93.541
135 100.000 100.000 99.276 97.118 93.657
136 100.000 100.000 99.297 97.180 93.772
137 100.000 100.000 99.317 97.242 93.888
138 100.000 100.000 99.338 97.300 93.992
139 100.000 100.000 99.359 97.358 94.096
140 100.000 100.000 99.379 97.416 94.199
141 100.000 100.000 99.400 97.473 94.304
142 100.000 100.000 99.421 97.531 94.407
143 100.000 100.000 99.441 97.589 94.510
144 100.000 100.000 99.462 97.643 94.603
145 100.000 100.000 99.483 97.697 94.695
146 100.000 100.000 99.503 97.750 94.788
147 100.000 100.000 99.524 97.804 94.880
148 100.000 100.000 99.545 97.858 94.972
149 100.000 100.000 99.565 97.912 95.065
150 100.000 100.000 99.586 97.966 95.157
151 100.000 100.000 99.607 98.019 95.250
152 100.000 100.000 99.628 98.073 95.342
153 100.000 100.000 99.648 98.127 95.434
154 100.000 100.000 99.669 98.181 95.527
155 100.000 100.000 99.690 98.234 95.619
156 100.000 100.000 99.710 98.288 95.712
157 100.000 100.000 99.731 98.342 95.804
158 100.000 100.000 99.752 98.396 95.896
159 100.000 100.000 99.772 98.449 95.989
160 100.000 100.000 99.793 98.503 96.081
161 100.000 100.000 99.814 98.557 96.174
162 100.000 100.000 99.834 98.611 96.266
163 100.000 100.000 99.855 98.664 96.358
164 100.000 100.000 99.876 98.718 96.451
165 100.000 100.000 99.897 98.772 96.543
166 100.000 100.000 99.917 98.826 96.636
167 100.000 100.000 99.938 98.879 96.728
168 100.000 100.000 99.959 98.933 96.820
169 100.000 100.000 99.979 98.987 96.913
170 100.000 100.000 100.000 99.041 97.005
171 100.000 100.000 100.000 99.065 97.075
172 100.000 100.000 100.000 99.090 97.144
173 100.000 100.000 100.000 99.107 97.193
174 100.000 100.000 100.000 99.123 97.241
175 100.000 100.000 100.000 99.140 97.290
176 100.000 100.000 100.000 99.148 97.315
177 100.000 100.000 100.000 99.156 97.341
178 100.000 100.000 100.000 99.165 97.366
179 100.000 100.000 100.000 99.173 97.392
180 100.000 100.000 100.000 99.181 97.417
181 100.000 100.000 100.000 99.190 97.443
182 100.000 100.000 100.000 99.198 97.469
183 100.000 100.000 100.000 99.206 97.494
184 100.000 100.000 100.000 99.214 97.520
185 100.000 100.000 100.000 99.223 97.545
186 100.000 100.000 100.000 99.231 97.571
187 100.000 100.000 100.000 99.239 97.596
188 100.000 100.000 100.000 99.247 97.622
189 100.000 100.000 100.000 99.256 97.647
190 100.000 100.000 100.000 99.264 97.673
191 100.000 100.000 100.000 99.272 97.698
192 100.000 100.000 100.000 99.280 97.724
193 100.000 100.000 100.000 99.289 97.749
194 100.000 100.000 100.000 99.297 97.775
195 100.000 100.000 100.000 99.305 97.800
196 100.000 100.000 100.000 99.314 97.826
197 100.000 100.000 100.000 99.322 97.852
198 100.000 100.000 100.000 99.330 97.877
199 100.000 100.000 100.000 99.338 97.903
200 100.000 100.000 100.000 99.347 97.928
201 100.000 100.000 100.000 99.355 97.954
202 100.000 100.000 100.000 99.363 97.979
203 100.000 100.000 100.000 99.371 98.005
204 100.000 100.000 100.000 99.380 98.030
205 100.000 100.000 100.000 99.388 98.056
206 100.000 100.000 100.000 99.396 98.081
207 100.000 100.000 100.000 99.405 98.107
208 100.000 100.000 100.000 99.413 98.132
209 100.000 100.000 100.000 99.421 98.158
210 100.000 100.000 100.000 99.429 98.183
211 100.000 100.000 100.000 99.438 98.209
212 100.000 100.000 100.000 99.446 98.235
213 100.000 100.000 100.000 99.454 98.260
214 100.000 100.000 100.000 99.462 98.286
215 100.000 100.000 100.000 99.471 98.311
216 100.000 100.000 100.000 99.479 98.337
217 100.000 100.000 100.000 99.487 98.362
218 100.000 100.000 100.000 99.496 98.388
219 100.000 100.000 100.000 99.504 98.413
220 100.000 100.000 100.000 99.512 98.439
221 100.000 100.000 100.000 99.520 98.464
222 100.000 100.000 100.000 99.529 98.490
223 100.000 100.000 100.000 99.537 98.515
224 100.000 100.000 100.000 99.545 98.541
225 100.000 100.000 100.000 99.553 98.566
226 100.000 100.000 100.000 99.562 98.592
227 100.000 100.000 100.000 99.570 98.618
228 100.000 100.000 100.000 99.578 98.643
229 100.000 100.000 100.000 99.586 98.669
230 100.000 100.000 100.000 99.595 98.694
231 100.000 100.000 100.000 99.603 98.720
232 100.000 100.000 100.000 99.611 98.745
233 100.000 100.000 100.000 99.620 98.771
234 100.000 100.000 100.000 99.628 98.796
235 100.000 100.000 100.000 99.632 98.810
236 100.000 100.000 100.000 99.636 98.823
237 100.000 100.000 100.000 99.640 98.836
238 100.000 100.000 100.000 99.644 98.850
239 100.000 100.000 100.000 99.649 98.863
240 100.000 100.000 100.000 99.653 98.877
241 100.000 100.000 100.000 99.657 98.890
242 100.000 100.000 100.000 99.661 98.903
243 100.000 100.000 100.000 99.665 98.917
244 100.000 100.000 100.000 99.669 98.930
245 100.000 100.000 100.000 99.673 98.943
246 100.000 100.000 100.000 99.677 98.957
247 100.000 100.000 100.000 99.682 98.970
248 100.000 100.000 100.000 99.686 98.984
249 100.000 100.000 100.000 99.690 98.997
250 100.000 100.000 100.000 99.694 99.010
251 100.000 100.000 100.000 99.698 99.024
252 100.000 100.000 100.000 99.702 99.037
253 100.000 100.000 100.000 99.706 99.050
254 100.000 100.000 100.000 99.711 99.064
255 100.000 100.000 100.000 99.715 99.077
256 100.000 100.000 100.000 99.719 99.091
257 100.000 100.000 100.000 99.723 99.104
258 100.000 100.000 100.000 99.727 99.117
259 100.000 100.000 100.000 99.731 99.131
260 100.000 100.000 100.000 99.735 99.144
261 100.000 100.000 100.000 99.739 99.157
262 100.000 100.000 100.000 99.744 99.171
263 100.000 100.000 100.000 99.748 99.184
264 100.000 100.000 100.000 99.752 99.198
265 100.000 100.000 100.000 99.756 99.211
266 100.000 100.000 100.000 99.760 99.224
267 100.000 100.000 100.000 99.764 99.238
268 100.000 100.000 100.000 99.768 99.251
269 100.000 100.000 100.000 99.773 99.264
270 100.000 100.000 100.000 99.777 99.278
271 100.000 100.000 100.000 99.781 99.291
272 100.000 100.000 100.000 99.785 99.305
273 100.000 100.000 100.000 99.789 99.318
274 100.000 100.000 100.000 99.793 99.331
275 100.000 100.000 100.000 99.797 99.345
276 100.000 100.000 100.000 99.802 99.358
277 100.000 100.000 100.000 99.806 99.371
278 100.000 100.000 100.000 99.810 99.385
279 100.000 100.000 100.000 99.814 99.398
280 100.000 100.000 100.000 99.818 99.412
281 100.000 100.000 100.000 99.822 99.425
282 100.000 100.000 100.000 99.826 99.438
283 100.000 100.000 100.000 99.830 99.452
284 100.000 100.000 100.000 99.835 99.465
285 100.000 100.000 100.000 99.839 99.478
286 100.000 100.000 100.000 99.843 99.492
287 100.000 100.000 100.000 99.847 99.505
288 100.000 100.000 100.000 99.851 99.519
289 100.000 100.000 100.000 99.855 99.532
290 100.000 100.000 100.000 99.859 99.545
291 100.000 100.000 100.000 99.864 99.559
292 100.000 100.000 100.000 99.868 99.572
293 100.000 100.000 100.000 99.872 99.585
294 100.000 100.000 100.000 99.876 99.599
295 100.000 100.000 100.000 99.880 99.612
296 100.000 100.000 100.000 99.884 99.626
297 100.000 100.000 100.000 99.888 99.639
298 100.000 100.000 100.000 99.892 99.652
299 100.000 100.000 100.000 99.897 99.666
300 100.000 100.000 100.000 99.901 99.679
301 100.000 100.000 100.000 99.905 99.692
302 100.000 100.000 100.000 99.909 99.706
303 100.000 100.000 100.000 99.913 99.719
304 100.000 100.000 100.000 99.917 99.733
305 100.000 100.000 100.000 99.921 99.746
306 100.000 100.000 100.000 99.926 99.759
307 100.000 100.000 100.000 99.930 99.773
308 100.000 100.000 100.000 99.934 99.786
309 100.000 100.000 100.000 99.938 99.799
310 100.000 100.000 100.000 99.942 99.813
311 100.000 100.000 100.000 99.946 99.826
312 100.000 100.000 100.000 99.950 99.840
313 100.000 100.000 100.000 99.955 99.853
314 100.000 100.000 100.000 99.959 99.866
315 100.000 100.000 100.000 99.963 99.880
316 100.000 100.000 100.000 99.967 99.893
317 100.000 100.000 100.000 99.971 99.906
318 100.000 100.000 100.000 99.975 99.920
319 100.000 100.000 100.000 99.979 99.933
320 100.000 100.000 100.000 99.983 99.947
321 100.000 100.000 100.000 99.988 99.960
322 100.000 100.000 100.000 99.992 99.973
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000

sed

Generation Strength: 5, Prioritisation Strength: 5

Results for test suite generated using x-way interaction coverage and prioritised by y-way interaction coverage. Coverage data show the rate of x-way interactions covered. To compare how quickly each prioritised test suite achieves the interaction coverage of a specific strength, we define an Average Percentage of Covering-array Coverage (APCC) metric following the standard Average Percentage of Fault Detection (APFD) metric. Given m covering arrays to cover and n test cases, let $CA_i$ be the index of the first test case that covers the interaction $CA$. APCC is defined as follows: $$APCC = (1 - \frac{\sum_i^m{CA_i}}{nm} + \frac{1}{2n}) \cdot 100$$

The rate at which faults have been detected for each version of the software is presented in the last plot, where 'max' denotes the number of seeded faults provided in SIR with each version.

Complete Results

Num. of Tests Cov. Str. t=1 Cov. Str. t=2 Cov. Str. t=3 Cov. Str. t=4 Cov. Str. t=5
1 37.500 17.442 9.414 5.644 3.651
2 55.000 30.233 17.670 10.999 7.234
3 72.500 43.355 26.154 16.458 10.854
4 75.000 49.668 32.050 20.986 14.150
5 80.000 55.482 37.120 24.997 17.202
6 85.000 60.299 41.299 28.446 19.950
7 87.500 64.120 45.107 31.737 22.624
8 92.500 69.103 49.245 35.025 25.203
9 95.000 72.757 52.742 37.981 27.593
10 97.500 75.748 55.576 40.471 29.703
11 100.000 78.571 58.245 42.840 31.736
12 100.000 80.066 60.315 44.949 33.658
13 100.000 80.897 61.866 46.768 35.433
14 100.000 82.060 63.563 48.575 37.141
15 100.000 83.389 65.405 50.465 38.884
16 100.000 84.718 67.225 52.318 40.582
17 100.000 85.714 68.694 53.902 42.101
18 100.000 86.711 70.163 55.473 43.590
19 100.000 87.708 71.591 56.986 45.031
20 100.000 87.708 72.212 58.041 46.247
21 100.000 88.704 73.619 59.513 47.633
22 100.000 89.535 74.840 60.836 48.917
23 100.000 90.199 75.874 62.011 50.099
24 100.000 90.698 76.723 63.032 51.169
25 100.000 91.196 77.529 63.991 52.182
26 100.000 91.860 78.523 65.087 53.265
27 100.000 92.359 79.350 66.067 54.281
28 100.000 92.857 80.178 67.047 55.296
29 100.000 93.023 80.654 67.762 56.141
30 100.000 93.189 81.130 68.478 56.986
31 100.000 93.688 81.916 69.388 57.923
32 100.000 94.518 83.054 70.545 58.987
33 100.000 94.850 83.675 71.335 59.846
34 100.000 95.183 84.295 72.125 60.704
35 100.000 95.515 84.916 72.915 61.562
36 100.000 95.681 85.351 73.560 62.330
37 100.000 95.847 85.785 74.201 63.086
38 100.000 96.013 86.220 74.842 63.842
39 100.000 96.179 86.634 75.446 64.555
40 100.000 96.346 87.027 76.012 65.223
41 100.000 96.512 87.420 76.574 65.881
42 100.000 96.678 87.792 77.108 66.516
43 100.000 96.844 88.185 77.666 67.162
44 100.000 97.010 88.579 78.224 67.807
45 100.000 97.176 88.972 78.783 68.453
46 100.000 97.342 89.365 79.341 69.098
47 100.000 97.508 89.717 79.837 69.689
48 100.000 97.508 89.882 80.193 70.199
49 100.000 97.674 90.213 80.656 70.756
50 100.000 98.007 90.730 81.260 71.392
51 100.000 98.339 91.248 81.863 72.027
52 100.000 98.671 91.765 82.467 72.663
53 100.000 98.837 92.117 82.955 73.231
54 100.000 99.003 92.468 83.443 73.799
55 100.000 99.169 92.800 83.898 74.333
56 100.000 99.336 93.131 84.353 74.867
57 100.000 99.502 93.462 84.808 75.400
58 100.000 99.668 93.793 85.262 75.934
59 100.000 99.668 94.000 85.655 76.439
60 100.000 99.668 94.206 86.048 76.943
61 100.000 99.668 94.413 86.441 77.448
62 100.000 99.834 94.744 86.887 77.961
63 100.000 99.834 94.869 87.156 78.351
64 100.000 100.000 95.179 87.566 78.821
65 100.000 100.000 95.303 87.826 79.188
66 100.000 100.000 95.427 88.087 79.555
67 100.000 100.000 95.531 88.310 79.878
68 100.000 100.000 95.634 88.533 80.202
69 100.000 100.000 95.738 88.757 80.525
70 100.000 100.000 95.800 88.914 80.783
71 100.000 100.000 95.862 89.067 81.031
72 100.000 100.000 95.945 89.249 81.300
73 100.000 100.000 96.027 89.431 81.568
74 100.000 100.000 96.110 89.613 81.837
75 100.000 100.000 96.193 89.794 82.106
76 100.000 100.000 96.276 89.976 82.375
77 100.000 100.000 96.358 90.158 82.643
78 100.000 100.000 96.420 90.311 82.889
79 100.000 100.000 96.483 90.464 83.135
80 100.000 100.000 96.545 90.617 83.380
81 100.000 100.000 96.607 90.770 83.626
82 100.000 100.000 96.669 90.923 83.871
83 100.000 100.000 96.731 91.076 84.117
84 100.000 100.000 96.793 91.229 84.363
85 100.000 100.000 96.855 91.382 84.608
86 100.000 100.000 96.917 91.535 84.854
87 100.000 100.000 96.979 91.688 85.099
88 100.000 100.000 97.041 91.841 85.345
89 100.000 100.000 97.103 91.994 85.591
90 100.000 100.000 97.165 92.147 85.836
91 100.000 100.000 97.227 92.300 86.082
92 100.000 100.000 97.289 92.449 86.319
93 100.000 100.000 97.352 92.594 86.544
94 100.000 100.000 97.414 92.739 86.769
95 100.000 100.000 97.476 92.883 86.994
96 100.000 100.000 97.538 93.028 87.219
97 100.000 100.000 97.600 93.173 87.444
98 100.000 100.000 97.662 93.318 87.669
99 100.000 100.000 97.724 93.462 87.893
100 100.000 100.000 97.765 93.574 88.084
101 100.000 100.000 97.827 93.710 88.286
102 100.000 100.000 97.890 93.847 88.488
103 100.000 100.000 97.952 93.983 88.690
104 100.000 100.000 98.014 94.120 88.892
105 100.000 100.000 98.076 94.256 89.094
106 100.000 100.000 98.138 94.393 89.295
107 100.000 100.000 98.200 94.529 89.497
108 100.000 100.000 98.262 94.666 89.699
109 100.000 100.000 98.324 94.802 89.901
110 100.000 100.000 98.386 94.939 90.103
111 100.000 100.000 98.448 95.075 90.305
112 100.000 100.000 98.510 95.212 90.506
113 100.000 100.000 98.572 95.348 90.708
114 100.000 100.000 98.634 95.484 90.910
115 100.000 100.000 98.696 95.621 91.112
116 100.000 100.000 98.759 95.757 91.314
117 100.000 100.000 98.821 95.894 91.516
118 100.000 100.000 98.862 95.997 91.686
119 100.000 100.000 98.903 96.101 91.856
120 100.000 100.000 98.945 96.196 92.003
121 100.000 100.000 98.986 96.291 92.150
122 100.000 100.000 99.007 96.353 92.266
123 100.000 100.000 99.028 96.415 92.381
124 100.000 100.000 99.048 96.477 92.497
125 100.000 100.000 99.069 96.539 92.612
126 100.000 100.000 99.090 96.601 92.728
127 100.000 100.000 99.110 96.663 92.843
128 100.000 100.000 99.131 96.725 92.959
129 100.000 100.000 99.152 96.787 93.074
130 100.000 100.000 99.172 96.849 93.190
131 100.000 100.000 99.193 96.911 93.305
132 100.000 100.000 99.214 96.973 93.421
133 100.000 100.000 99.234 97.035 93.536
134 100.000 100.000 99.255 97.097 93.652
135 100.000 100.000 99.276 97.159 93.767
136 100.000 100.000 99.297 97.221 93.883
137 100.000 100.000 99.317 97.283 93.998
138 100.000 100.000 99.338 97.345 94.114
139 100.000 100.000 99.359 97.403 94.218
140 100.000 100.000 99.379 97.461 94.323
141 100.000 100.000 99.400 97.519 94.426
142 100.000 100.000 99.421 97.577 94.530
143 100.000 100.000 99.441 97.631 94.622
144 100.000 100.000 99.462 97.684 94.715
145 100.000 100.000 99.483 97.738 94.807
146 100.000 100.000 99.503 97.792 94.899
147 100.000 100.000 99.524 97.846 94.992
148 100.000 100.000 99.545 97.899 95.084
149 100.000 100.000 99.565 97.953 95.177
150 100.000 100.000 99.586 98.007 95.269
151 100.000 100.000 99.607 98.061 95.361
152 100.000 100.000 99.628 98.114 95.454
153 100.000 100.000 99.648 98.168 95.546
154 100.000 100.000 99.669 98.222 95.639
155 100.000 100.000 99.690 98.276 95.731
156 100.000 100.000 99.710 98.329 95.823
157 100.000 100.000 99.731 98.383 95.916
158 100.000 100.000 99.752 98.437 96.008
159 100.000 100.000 99.772 98.491 96.101
160 100.000 100.000 99.793 98.544 96.193
161 100.000 100.000 99.814 98.598 96.285
162 100.000 100.000 99.834 98.652 96.378
163 100.000 100.000 99.855 98.706 96.470
164 100.000 100.000 99.876 98.759 96.563
165 100.000 100.000 99.897 98.813 96.655
166 100.000 100.000 99.917 98.867 96.748
167 100.000 100.000 99.938 98.921 96.840
168 100.000 100.000 99.959 98.974 96.932
169 100.000 100.000 99.979 99.028 97.025
170 100.000 100.000 100.000 99.082 97.117
171 100.000 100.000 100.000 99.099 97.166
172 100.000 100.000 100.000 99.115 97.214
173 100.000 100.000 100.000 99.132 97.263
174 100.000 100.000 100.000 99.140 97.289
175 100.000 100.000 100.000 99.148 97.314
176 100.000 100.000 100.000 99.156 97.340
177 100.000 100.000 100.000 99.165 97.365
178 100.000 100.000 100.000 99.173 97.391
179 100.000 100.000 100.000 99.181 97.416
180 100.000 100.000 100.000 99.190 97.442
181 100.000 100.000 100.000 99.198 97.467
182 100.000 100.000 100.000 99.206 97.493
183 100.000 100.000 100.000 99.214 97.518
184 100.000 100.000 100.000 99.223 97.544
185 100.000 100.000 100.000 99.231 97.569
186 100.000 100.000 100.000 99.239 97.595
187 100.000 100.000 100.000 99.247 97.621
188 100.000 100.000 100.000 99.256 97.646
189 100.000 100.000 100.000 99.264 97.672
190 100.000 100.000 100.000 99.272 97.697
191 100.000 100.000 100.000 99.280 97.723
192 100.000 100.000 100.000 99.289 97.748
193 100.000 100.000 100.000 99.297 97.774
194 100.000 100.000 100.000 99.305 97.799
195 100.000 100.000 100.000 99.314 97.825
196 100.000 100.000 100.000 99.322 97.850
197 100.000 100.000 100.000 99.330 97.876
198 100.000 100.000 100.000 99.338 97.901
199 100.000 100.000 100.000 99.347 97.927
200 100.000 100.000 100.000 99.355 97.952
201 100.000 100.000 100.000 99.363 97.978
202 100.000 100.000 100.000 99.371 98.004
203 100.000 100.000 100.000 99.380 98.029
204 100.000 100.000 100.000 99.388 98.055
205 100.000 100.000 100.000 99.396 98.080
206 100.000 100.000 100.000 99.405 98.106
207 100.000 100.000 100.000 99.413 98.131
208 100.000 100.000 100.000 99.421 98.157
209 100.000 100.000 100.000 99.429 98.182
210 100.000 100.000 100.000 99.438 98.208
211 100.000 100.000 100.000 99.446 98.233
212 100.000 100.000 100.000 99.454 98.259
213 100.000 100.000 100.000 99.462 98.284
214 100.000 100.000 100.000 99.471 98.310
215 100.000 100.000 100.000 99.479 98.335
216 100.000 100.000 100.000 99.487 98.361
217 100.000 100.000 100.000 99.496 98.387
218 100.000 100.000 100.000 99.504 98.412
219 100.000 100.000 100.000 99.512 98.438
220 100.000 100.000 100.000 99.520 98.463
221 100.000 100.000 100.000 99.529 98.489
222 100.000 100.000 100.000 99.537 98.514
223 100.000 100.000 100.000 99.545 98.540
224 100.000 100.000 100.000 99.553 98.565
225 100.000 100.000 100.000 99.562 98.591
226 100.000 100.000 100.000 99.570 98.616
227 100.000 100.000 100.000 99.578 98.642
228 100.000 100.000 100.000 99.586 98.667
229 100.000 100.000 100.000 99.595 98.693
230 100.000 100.000 100.000 99.603 98.718
231 100.000 100.000 100.000 99.611 98.744
232 100.000 100.000 100.000 99.620 98.770
233 100.000 100.000 100.000 99.624 98.783
234 100.000 100.000 100.000 99.628 98.796
235 100.000 100.000 100.000 99.632 98.810
236 100.000 100.000 100.000 99.636 98.823
237 100.000 100.000 100.000 99.640 98.836
238 100.000 100.000 100.000 99.644 98.850
239 100.000 100.000 100.000 99.649 98.863
240 100.000 100.000 100.000 99.653 98.877
241 100.000 100.000 100.000 99.657 98.890
242 100.000 100.000 100.000 99.661 98.903
243 100.000 100.000 100.000 99.665 98.917
244 100.000 100.000 100.000 99.669 98.930
245 100.000 100.000 100.000 99.673 98.943
246 100.000 100.000 100.000 99.677 98.957
247 100.000 100.000 100.000 99.682 98.970
248 100.000 100.000 100.000 99.686 98.984
249 100.000 100.000 100.000 99.690 98.997
250 100.000 100.000 100.000 99.694 99.010
251 100.000 100.000 100.000 99.698 99.024
252 100.000 100.000 100.000 99.702 99.037
253 100.000 100.000 100.000 99.706 99.050
254 100.000 100.000 100.000 99.711 99.064
255 100.000 100.000 100.000 99.715 99.077
256 100.000 100.000 100.000 99.719 99.091
257 100.000 100.000 100.000 99.723 99.104
258 100.000 100.000 100.000 99.727 99.117
259 100.000 100.000 100.000 99.731 99.131
260 100.000 100.000 100.000 99.735 99.144
261 100.000 100.000 100.000 99.739 99.157
262 100.000 100.000 100.000 99.744 99.171
263 100.000 100.000 100.000 99.748 99.184
264 100.000 100.000 100.000 99.752 99.198
265 100.000 100.000 100.000 99.756 99.211
266 100.000 100.000 100.000 99.760 99.224
267 100.000 100.000 100.000 99.764 99.238
268 100.000 100.000 100.000 99.768 99.251
269 100.000 100.000 100.000 99.773 99.264
270 100.000 100.000 100.000 99.777 99.278
271 100.000 100.000 100.000 99.781 99.291
272 100.000 100.000 100.000 99.785 99.305
273 100.000 100.000 100.000 99.789 99.318
274 100.000 100.000 100.000 99.793 99.331
275 100.000 100.000 100.000 99.797 99.345
276 100.000 100.000 100.000 99.802 99.358
277 100.000 100.000 100.000 99.806 99.371
278 100.000 100.000 100.000 99.810 99.385
279 100.000 100.000 100.000 99.814 99.398
280 100.000 100.000 100.000 99.818 99.412
281 100.000 100.000 100.000 99.822 99.425
282 100.000 100.000 100.000 99.826 99.438
283 100.000 100.000 100.000 99.830 99.452
284 100.000 100.000 100.000 99.835 99.465
285 100.000 100.000 100.000 99.839 99.478
286 100.000 100.000 100.000 99.843 99.492
287 100.000 100.000 100.000 99.847 99.505
288 100.000 100.000 100.000 99.851 99.519
289 100.000 100.000 100.000 99.855 99.532
290 100.000 100.000 100.000 99.859 99.545
291 100.000 100.000 100.000 99.864 99.559
292 100.000 100.000 100.000 99.868 99.572
293 100.000 100.000 100.000 99.872 99.585
294 100.000 100.000 100.000 99.876 99.599
295 100.000 100.000 100.000 99.880 99.612
296 100.000 100.000 100.000 99.884 99.626
297 100.000 100.000 100.000 99.888 99.639
298 100.000 100.000 100.000 99.892 99.652
299 100.000 100.000 100.000 99.897 99.666
300 100.000 100.000 100.000 99.901 99.679
301 100.000 100.000 100.000 99.905 99.692
302 100.000 100.000 100.000 99.909 99.706
303 100.000 100.000 100.000 99.913 99.719
304 100.000 100.000 100.000 99.917 99.733
305 100.000 100.000 100.000 99.921 99.746
306 100.000 100.000 100.000 99.926 99.759
307 100.000 100.000 100.000 99.930 99.773
308 100.000 100.000 100.000 99.934 99.786
309 100.000 100.000 100.000 99.938 99.799
310 100.000 100.000 100.000 99.942 99.813
311 100.000 100.000 100.000 99.946 99.826
312 100.000 100.000 100.000 99.950 99.840
313 100.000 100.000 100.000 99.955 99.853
314 100.000 100.000 100.000 99.959 99.866
315 100.000 100.000 100.000 99.963 99.880
316 100.000 100.000 100.000 99.967 99.893
317 100.000 100.000 100.000 99.971 99.906
318 100.000 100.000 100.000 99.975 99.920
319 100.000 100.000 100.000 99.979 99.933
320 100.000 100.000 100.000 99.983 99.947
321 100.000 100.000 100.000 99.988 99.960
322 100.000 100.000 100.000 99.992 99.973
323 100.000 100.000 100.000 99.996 99.987
324 100.000 100.000 100.000 100.000 100.000