Skip to content

Exercise 2: Combinatorial Testing with ACTS, Control Flow Graph, & AI Test Evaluation

In this exercise, you will practice model-based test design, combinatorial test generation, and control flow analysis on a fork of Spring Petclinic REST. You will apply functionality-based Input Domain Modeling (IDM) using the NIST ACTS (Automated Combinatorial Testing for Software) tool to generate a Pair-Wise Coverage (PWC) test suite, construct a Control Flow Graph (CFG) to satisfy Edge-Pair Coverage (EPC), build an automated data-driven test script in JUnit 5, and critically evaluate an AI coding assistant's test generation capabilities against formal combinatorial models.

Getting Started

Complete the following steps before starting the exercise:

  1. Ensure you are enrolled in the SQA (PMPL) course and have an active account on GitLab CSUI.
  2. Set up the required tools on your local development machine:
  3. Download the NIST ACTS tool archive from SCELE.
    • You can also download from NIST ACTS website for the basic version. However, SCELE provides the latest, advanced version of NIST ACTS.
  4. Verify that you can launch ACTS on your machine (for example, by running java -jar acts.jar).
  5. In your local clone of your spring-petclinic-rest repository (from Exercise 1): 
    1
2
    git switch csui
git pull origin csui
  6. Create a new working branch for this exercise: 
    1
    git switch -c task/2-modeling

Exercise Scope

You will analyse two methods from different layers of the codebase:

  1. Owner.getPet:
    • It is a method that implements domain logic for traversing and filtering a collection of Pet entities.
    • Location: org.springframework.samples.petclinic.model.Owner
    • Signature: public Pet getPet(String name, boolean ignoreNew)
  2. PetAgeValidator.isValid:
    • It is a method that provides validation logic for checking date boundaries and business constraint.
    • Location: org.springframework.samples.petclinic.rest.validation.PetAgeValidator
    • Signature: public boolean isValid(LocalDate birthDate, ConstraintValidatorContext context)

Task 1: Functionality-Based Input Domain Modeling (IDM) with ACTS

Do the following steps for both Owner.getPet and PetAgeValidator.isValid methods:

  1. Identify characteristics and partitions (blocks)
    • For Owner.getPet: Consider semantic partitions such as:
      • Target name match condition (e.g., exact match, case variation, non-existent name, empty/null string).
      • Parameter ignoreNew flag (true, false).
      • State of owner's pet collection (e.g., empty collection, collection containing only newly created pets, collection containing saved/persisted pets, collection with multiple pets).
    • For PetAgeValidator.isValid: Consider semantic partitions such as:
      • birthDate category (e.g., null, date in the future, date within valid age range).
      • context presence (e.g., null, active mock context).
    • Ensure the blocks for each characteristic are disjoint and complete.
  2. Formulate constraints to eliminate unfeasible or invalid combinations.
  3. Build the model in NIST ACTS. Define the system (SUT), parameters, and constraints. Then, configure the combinatorial coverage criterion to 2-way strength, which corresponds to Pair-Wise Coverage. Lastly, run the generation algorithm (IPOG) to create a minimal covering array.
  4. Export deliverables by saving the ACTS model configuration as XML file and the generated test suite configuration as CSV file.

Task 2: Data-Driven Automated Testing for PetAgeValidator

Take a look at the generated test suite configuration file. You will notice that NIST ACTS only generates the test inputs based on the model. The model itself does not have the knowledge of the expected outcomes for each generated test input.

In this task, you will augment the generated inputs with the expected outcomes so it can be fed into a data-driven test script:

  1. Update the ACTS CSV with a new column for the expected outcome. Open your exported ACTS CSV for PetAgeValidator. Then, add a header and a new column named expectedIsValid representing the expected outcome.

  2. Implement parameterised test in the project. Use JUnit 5's @ParameterizedTest with @CsvFileSource to read the CSV file, initialize test fixtures, execute validator.isValid(...), and assert against expectedIsValid.

    • Use the following reference implementation largely generated by GenAI as a guide:
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
    // SPDX-License-Identifier: CC0-1.0
// Assisted-by: Gemini 3.8 Flash
package org.springframework.validation;

import jakarta.validation.ConstraintValidatorContext;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvFileSource;
import org.springframework.samples.petclinic.rest.validation.PetAgeValidator;

import java.time.LocalDate;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

class PetAgeValidatorDataDrivenTest {

    private final PetAgeValidator validator = new PetAgeValidator();

    @ParameterizedTest(name = "[{index}] {0}: yearOffset={1}, hasContext={2}, expected={3}")
    @CsvFileSource(resources = "/pet_age_validator_pwc.csv", numLinesToSkip = 1)
    void testPetAgeValidatorPWC(String testCaseId, String yearOffsetStr, boolean hasContext, boolean expectedIsValid) {
        LocalDate birthDate = "null".equalsIgnoreCase(yearOffsetStr) || yearOffsetStr == null
                ? null
                : LocalDate.now().plusYears(Long.parseLong(yearOffsetStr));

        ConstraintValidatorContext context = null;
        if (hasContext) {
            context = mock(ConstraintValidatorContext.class);
            when(context.buildConstraintViolationWithTemplate(anyString()))
                    .thenReturn(mock(ConstraintValidatorContext.ConstraintViolationBuilder.class));
        }

        boolean actual = validator.isValid(birthDate, context);
        assertEquals(expectedIsValid, actual, "Assertion failed for test case: " + testCaseId);
    }
}

  3. Execute the data-driven test suite. You can run it via your IDE (e.g., IntelliJ) or via terminal:

    1
    ./mvnw test -Dtest=PetAgeValidatorDataDrivenTest
    Ensure all test cases derived from your ACTS model execute cleanly and pass.

Task 3: Control Flow Graph (CFG) & Edge-Pair Coverage (EPC)

In this task, you will analyse the execution paths of both target methods using graph modeling.

For both Owner.getPet and PetAgeValidator.isValid:

  1. Create the CFG using a diagramming tool (e.g., Draw.io, Excalidraw, PlantUML). Do not forget to label each node with a number or symbol, and label the outward edge(s) from each branching node with the conditional expression.
  2. Enumerate test requirements using EPC criterion.
  3. Derive the minimal test paths to cover the test requirements. Label any unfeasible test path, if any.

Task 4: Generative AI Test Generation Comparison & Reflection

Generative AI (GenAI)-based assistants that run on Web browser (e.g., ChatGPT) or directly in development environment (e.g., GitHub Copilot) can generate unit test suites in seconds. However, how well do AI-generated tests satisfy the test requirements derived from systematic combinatorial and graph-based models built with ACTS and CFGs?

To evaluate this, you will prompt a GenAI assistant to generate the unit test suite without prior knowledge of existing test suites in the codebase and without knowledge about this exercise specifications. The assistant should generate the tests as if there were no tests in the codebase and it has not read this document.

1. Generate Tests via Prompt in Isolation

Copy and paste the following prompt template into a standalone AI chat session on a Web-based chat (e.g., Claude, ChatGPT), or an isolated AI agent session with no access to the source code:

Prompt template for PetAgeValidator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
You are a senior Software Engineer in Test.
Help me to write a comprehensive suite of JUnit 5 unit tests for the following Java validator method.
Focus on boundary values, null inputs, edge cases, and mocking dependencies where necessary.

package org.springframework.samples.petclinic.rest.validation;

import java.time.LocalDate;
import jakarta.validation.ConstraintValidator;
import jakarta.validation.ConstraintValidatorContext;

public class PetAgeValidator implements ConstraintValidator<PetAgeValidation, LocalDate> {

    @Override
    public boolean isValid(LocalDate birthDate, ConstraintValidatorContext context) {
        if (birthDate == null) {
            return true; 
        }

        LocalDate today = LocalDate.now();

        if (birthDate.isAfter(today)) {
            context.disableDefaultConstraintViolation();
            context.buildConstraintViolationWithTemplate(
                "Birth date cannot be in the future"
            ).addConstraintViolation();
            return false;
        }

        if (birthDate.isBefore(today.minusYears(50))) {
            context.disableDefaultConstraintViolation();
            context.buildConstraintViolationWithTemplate(
                "Birth date cannot be older than 50 years"
            ).addConstraintViolation();
            return false;
        }

        return true;
    }
}

(Optional: You may also run a similar prompt for Owner.getPet).

2. Compare AI Output Against IDM & CFG

Critically compare the AI-generated test suite against:

  • ACTS PWC Matrix: Did the AI cover all combinations of inputs? Did it test the exact boundary date of 50 years ago, or only generic values like 10 years and 60 years?
  • CFG EPC Test Paths: Did the AI exercise all edge pairs? Did it detect the risk of NullPointerException if context is null?

Document your evaluation and reflection (2–3 paragraphs) in your written report.

Deliverables

At the end of this exercise, commit and push the following artifacts to your fork on GitLab CSUI under branch task/2-modeling (no need to merge the new files into the main branch):

  1. Written Report: A Markdown file named EXERCISE_2_REPORT.md in the repository root.
  2. ACTS Model & Data Files:
    • ACTS model files (.xml) for both methods.
    • Exported CSV files from ACTS.
    • The augmented CSV file used for data-driven testing.
  3. CFG Diagrams: Exported image files (PNG or JPEG) referenced within your report.
  4. Codebase Updates:
    • src/test/resources/pet_age_validator_pwc.csv
    • src/test/java/org/springframework/validation/PetAgeValidatorDataDrivenTest.java
    • All tests passing under ./mvnw test.

The due date of this exercise is: Sunday, 22 November 2026, 23:55 UTC+7.

Submit the URL of your fork repository to the designated submission slot on SCELE before the deadline.

GenAI Usage Policy

This exercise allows GenAI and LLM tools as learning materials and tools. Even so, it is still imperative for you as course participant of SQA and user of GenAI to factually check the responses generated by GenAI.

The use of GenAI does not mean you can entirely delegate your learning process to GenAI. You enrolled in a course to learn and obtain knowledge and skills. Do not make GenAI the one that does the learning, turning you into a "meat proxy" for it.

Allowed Uses

  • Conducting the comparative evaluation required in Task 4 using the provided prompt.
  • Asking for docs and snippets regarding JUnit 5 @ParameterizedTest or ACTS configuration.
  • Brainstorming potential semantic partitions.
  • Directing assistants to execute test suites and self-correct based on test feedback.

Prohibited Uses

  • Direct, uncritical copy-pasting of AI-generated tables, graphs, or calculations without verification.
  • Outsourcing the construction of the CFG or IDM partitions entirely to an AI tool.

Report Template

Use the following Markdown template for EXERCISE_2_REPORT.md:

Note: you can write your report in English or Bahasa Indonesia.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# Exercise 2 Report: Combinatorial Testing with ACTS & CFG

- **Name:** <NAME>
- **NPM:** <NPM>
- **GitLab Repository URL:** <URL>

## 1. Input Domain Modeling with NIST ACTS

### Method 1: `Owner.getPet`

#### Functionality-Based Characteristics & Partitions

> The following table is only an example.
> Do not forget to adapt to the case you are working on.

| Characteristic | Description | Partition Blocks |
| :--- | :--- | :--- |
| `Distinction` | Possible distinction title given during graduation | `[A1 (SUMMA_CUM_LAUDE), A2 (CUM_LAUDE), A3 (SANGAT_MEMUASKAN), A4 (MEMUASKAN)]` |
| `Retake` | Does a student ever retake a course? | `[B1 (T), B2 (F)]` |
| `Study Duration` | How long was the student actively enrolled until graduation? | `[C1 (LESS_THAN_8_SEM), C2 (EXACT_8_SEM), C3 (MORE_THAN_8_SEM)]` |

#### Constraints

- `<Enumerate constraints defined in ACTS, if any>`

#### ACTS Pair-Wise Coverage (PWC) Results

- **ACTS Configuration:** Degree of Interaction = 2, Algorithm = IPOG
- **Number of Generated Test Configurations:** `<N>`
- `<Attach or summarize generated table>`

### Method 2: `PetAgeValidator.isValid`

> Follow similar structure like Method 1 above.

## 2. Data-Driven Testing for `PetAgeValidator`

- **CSV File Location:** `<Location of the CSV file updated with expected outcome for each test input>`
- **Test Class:** `<Location of the new or updated Java test suite class that runs the data-driven test>`
- **Test Execution Result:** `<Attach snippet of passing test run from ./mvnw test>`

## 3. Control Flow Graph (CFG) & Edge-Pair Coverage (EPC)

### Method 1: `Owner.getPet`

- **CFG Diagram:** `![Owner CFG](images/owner_getpet_cfg.png)`
- **Edge-Pair Requirements:** `<List of edge-pairs>`
- **Test Paths:** `<Enumerate test paths and label whether each path is feasible or not>`

### Method 2: `PetAgeValidator.isValid`

> Follow similar structure like Method 1 above.

## 4. AI Test Generation Comparison & Reflection

- **AI Tool Used:** `<e.g., ChatGPT / Claude / Copilot Chat in isolation>`
- **Comparison Against ACTS PWC Matrix:**
  `<Discuss which combinations and boundary values are missed by the AI-generated tests.>`
- **Comparison Against CFG EPC Paths:**
  `<Discuss which control paths, loop edge cases, or exception paths are missed by the AI-generated tests.>`
- **Reflection (2–3 paragraphs):**
  `<Critique of LLM capabilities vs formal model-based test design.>`

## 5. Interesting/Relevant Findings (optional)

> Write down your interesting findings here, if any, that not necessarily related to the exercise but still relevant to the SQA as a whole.

GenAI Use Disclosure

Antigravity CLI with Gemini 3.8 Flash model was used to:

  • Proofread this document
  • Extract the PetAgeValidator method snippet from the codebase
  • Generate an example of data-driven test code
  • Generate the report template

We claim no authorship of the data-driven test code example. Hence, we release the code under CC0 1.0 Universal (public domain). We have reviewed and edited the generated text.

References

  • NIST Automated Combinatorial Testing for Software (ACTS): https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software
  • Paul Ammann & Jeff Offutt, Introduction to Software Testing, 2nd Edition (Cambridge University Press).
  • Chapter 6: Input Space Partitioning
  • Chapter 7: Graph Coverage
  • JUnit 5 User Guide: Parameterized Tests: https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests
Last update: 2026-11-14 01:30:54
Created: 2026-11-14 01:30:54