Exercise 4: Mutation Testing

Mutation Testing

Mutation testing is a test that involves modifying a program in small ways, usually by changing a variable or an operator. The goal here is to "kill" the modified program with test cases, that is, by rejecting the mutant using the original program's test cases.

Creating Mutants

The mutant is created by applying one of these operation on a program line.

Absolute Value Insertion (ABS): Each arithmetic expression (and subexpression) is modified by the functions abs(), negAbs(), and failOnZero().

    a = m * (o + p);
∆1  a = abs (m * (o + p));
∆2  a = m * abs ((o + p));
∆3  a = failOnZero (m * (o + p));

Arithmetic Operator Replacement (AOR): Each occurrence of one of the arithmetic operators is replaced by each of the other operators. It can also be replaced by the special mutation operators such as leftOp and rightOp.
1 2 3 4
a = m * (o + p); ∆1 a = m + (o + p); ∆2 a = m * (o * p); ∆3 a = m leftOp (o + p);
Relational Operator Replacement (ROR): Each occurrence of one of the relational operators (<, ≤, >, ≥, =, ≠) is replaced by each of the other operators and by falseOp and trueOp.
1 2 3 4
if (X <= Y) ∆1 if (X > Y) ∆2 if (X < Y) ∆3 if (X falseOp Y) // always returns false
Conditional Operator Replacement (COR): Each occurrence of one of the logical operators (&&, ||, etc.) is replaced by each of the other operators. In addition, each is replaced by falseOp, trueOp, leftOp, and rightOp.
1 2 3
if (X <= Y && a > 0) ∆1 if (X <= Y || a > 0) ∆2 if (X <= Y leftOp a > 0) // returns result of left clause
Shift Operator Replacement (SOR): Each occurrence of one of the shift operators <<, >>, and >>> is replaced by each of the other operators. In addition, each is replaced by the special mutation operator leftOp.
1 2 3 4
byte b = (byte) 16; b = b >> 2; ∆1 b = b << 2; ∆2 b = b leftOp 2; // result is b
Logical Operator Replacement (LOR): Each occurrence of one of the logical operators (bitwise and - &, bitwise or - |, exclusive or - ^) is replaced by each of the other operators. In addition, each is replaced by leftOp and rightOp.
1 2 3 4
int a = 60; int b = 13; int c = a & b; ∆1 int c = a | b; ∆2 int c = a rightOp b; // result is b
Assignment Operator Replacement (ASR): Each occurrence of one of the assignment operators (=, +=, -=, etc.) is replaced by each of the other operators.
1 2 3
a = m * (o + p); ∆1 a += m * (o + p); ∆2 a *= m * (o + p);
Unary Operator Insertion (UOI): Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is inserted in front of each expression of the correct type.
1 2 3
a = m * (o + p); ∆1 a = m * -(o + p); ∆2 a = -(m * (o + p));
Unary Operator Deletion (UOD): Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is deleted.
1 2 3
if !(X <= Y && !Z) ∆1 if (X > Y && !Z) ∆2 if !(X < Y && Z)

Scalar Variable Replacement (SVR): Each variable reference is replaced by every other variable of the appropriate type that is declared in the current scope.

    a = m * (o + p);
∆1  a = o * (o + p);
∆2  a = m * (m + p);
∆3  a = m * (o + o);
∆4  p = m * (o + p);

Bomb Statement Replacement (BSR): Each statement is replaced by a special Bomb() function.
1 2
a = m * (o + p); ∆1 Bomb() // Raises exception when reached

Killing Mutants

To kill a mutant, you need to recall the RIPR model introduced on Introduction to Software Testing by Paul Amann and Jeff Offutt. Chapter 2:

Reachability: The test causes the faulty statement to be reached (in mutation – the mutated statement).
Infection: The test causes the faulty statement to result in an incorrect state.
Propagation: The incorrect state propagates to incorrect output.
Revealability : The tester must observe part of the incorrect output.

The RIPR model leads to two variants of mutant killing:

Weakly killing mutants: create a test that kill mutants with incorrect state. To achieve this, you need to create a test that satisfies reachability and infection, but not propagation.
Strongly killing mutants: create a test that kill mutants with incorrect output. To achieve this, you need to create a test that satisfies reachability, infection, and propagation.

To demonstrate how we kill a mutant, take a look at the snippet below:

boolean isEven (int X) {
    if (X < 0)
        X = 0 - X;
∆       X = 0;
    if (double) (X/2) == ((double) X) / 2.0
        return (true);
    else
        return (false);
    }

Ignoring the revealability, we can summarise our RIPR model as follows:

Reachability (R): X < 0
Infection (I): X != 0
Propagation (P): odd(0-X)

Now that we have found the reachability, infection, and propagation of the mutant, we can finally create a full test specification to kill the mutant.

To weakly kill the mutant, we need to satisfy these conditions: R && I && !P.

Weak kill = R && I && !P
          = (X < 0) && (X != 0) && !odd(0-X)
          = (X < 0) && even(0-X)

One of the X value that satifies this is X=-6.

To strongly kill the mutant, we need to satisfy these conditions: R && I && P.

Weak kill = R && I && P
          = (X < 0) && (X != 0) && odd(0-X)
          = (X < 0) && odd(0-X)

One of the X value that satifies this is X=-5.

For more information regarding mutant killings, please read the Introduction to Software Testing by Paul Amann and Jeff Offutt on Chapter 9.2.2

Tasks

You are asked to create a mutation test on your group project individually. Implement this exercise using a new branch of your previously forked group project codebase.

This exercise consist of 2 phases, where each phase requires you to commit and push your work to GitLab CS.

At the end of the exercise, do not forget to schedule an one-on-one meeting with a teaching assistant to demonstrate your work.

Task 1: Creating Mutants

You have learnt about how mutation testing works. Now it is your turn to conduct it in your forked repository. Please do these tasks below.

Pick one complex method that is different from what you use on previous exercise, and then try to create at least 2 mutants from it with 2 different kinds of mutations. You can write the mutants on code or in a written document.
Create the full test specification to strongly kill your mutants in a written document by analyzing the reachability, infection, and propagation of those mutants.
Compare the full test specification with the current method's test suite? Are they current test suite enough to kill all mutants?

You can relate question 2 and 3 with your full test specification for each mutants.

Task 2: Mutation Testing Tools

There are several tools that can support mutation testing. Try to use one of them by doing these tasks below.

Choose one mutation testing tools and apply it on your forked repository.
Use that tool to conduct mutation testing on your selected method (not the mutants you implement).
Improve your test based on the conducted mutation testing.
Modify your .gitlab-ci.yml to include mutation testing using the selected tool if necessary.
Commit and push your improvement.

After conducting all the tasks above, please explain these in a written document:

Deliverables

At the end of this exercise, you are required to prepare the following artifacts:

A written document that describes the process of your work in completing this exercise. You can write the document as a Markdown-formatted text file or a PDF file. Give the document a descriptive name, e.g. exercise4.md, and put it into a folder called docs in your fork.
The mutants you created.
One or more changed test suites, if any, that had been updated from the result of mutation testing tools.

If there are no changes in the test suites, explain the reasons in the documentation.

Work Demonstration

Arrange an one-on-one meeting with a teaching assistant to demonstrate your work. You are expected to be able to:

Describe how you conduct your full test specification and how it can kill your mutants.
Explain about the mutation testing tools you're using.
Explain the test improvement using mutation testing tools.
Explain the benefit of mutatin testing.

The due date of this exercise is: 8 December 2021, 21:00 UTC+7. Please ensure any updates to the fork repository related to this exercise were made and pushed before the due date.

References

Introduction to Software Testing by Paul Amann and Jeff Offutt.
Mutation Testing Tools for Java

Last update: 2021-12-01 13:24:49
Created: 2021-12-01 13:24:49