Skip to content

Project 2: Combinatorial & Model-Based Testing on Real-World Projects

This assignment is a continuation of Exercise 2 and contributes to the project grade component. In this project, you will apply functionality-based Input Domain Modeling (IDM) using NIST ACTS and Control Flow Graph (CFG) modeling with Edge-Pair Coverage (EPC) to the real-world codebase you selected and onboarded in Project 1.

Like Project 1, this project is an individual assignment. Every student is independently responsible for selecting a non-trivial method in their own repository, deriving combinatorial test requirements and control paths, implementing automated tests in the project's native test framework, maintaining a passing SonarQube Quality Gate, and critically reflecting on AI-assisted test generation against systematic models.

Target Method Selection Criteria

You will select one non-trivial method from the codebase you used in Project 1. The selected method must satisfy the following criteria:

  1. Rich Domain Logic: The method must have meaningful business or domain logic (avoid trivial getters/setters, one-line delegate wrappers, or plain CRUD passthroughs).
  2. Multiple Semantic Parameters / States: The method must accept multiple input arguments, complex object parameters, or depend on non-trivial internal state suitable for partitioning.
  3. Branching & Control Flow: The method must contain multiple conditional decisions (if/else, switch, guard clauses), loops (for, while), or exception paths suitable for CFG edge-pair derivation.
  4. Unit-Testability: You must be able to write and execute automated unit tests against this method within your project's existing test framework (e.g., JUnit 5, pytest, Jest/Vitest, Go test).

Getting Started

  1. In your local repository for your Project 1 codebase: 
    1
2
    git switch main
git pull origin main
    (Note: If your default branch is named master or something else, use that branch name).
  2. Ensure that your test suite passes cleanly locally and that your GitLab CI/CD pipeline is green.
  3. Create a working branch for this project: 
    1
    git switch -c task/2-modeling

Task 1: Functionality-Based Input Domain Modeling (IDM) with NIST ACTS

Apply functionality-based IDM to your chosen target method:

  1. Identify characteristics and partitions (blocks).
    • Focus strictly on functionality-based characteristics (domain semantics, input ranges, state invariants, boundary values). Avoid superficial interface-based syntax checks.
    • Ensure the blocks for each characteristic are disjoint and complete.
  2. Formulate constraints to eliminate unfeasible or invalid combinations.
  3. Build the model in NIST ACTS. Define the system (SUT), parameters, and constraints. Then, configure the combinatorial coverage criterion to 2-way strength, which corresponds to Pair-Wise Coverage (PWC). Lastly, run the generation algorithm (IPOG) to create a minimal covering array.
  4. Export deliverables by saving the ACTS model configuration as XML or text file and the generated test suite configuration as CSV file.

Task 2: Control Flow Graph (CFG) & Edge-Pair Coverage (EPC)

In this task, you will analyse the execution paths of your chosen target method using graph modeling:

  1. Create the CFG using a diagramming tool (e.g., Draw.io, Excalidraw, PlantUML, Mermaid). Do not forget to label each node with a number or symbol, and label the outward edge(s) from each branching node with the conditional expression.
  2. Enumerate test requirements using EPC criterion.
  3. Derive the minimal test paths to cover the test requirements. Label any unfeasible test path, if any.

Task 3: Automated Test Implementation & Quality Gate Maintenance

Implement the test requirements derived from ACTS and your CFG into your codebase's automated test suite:

  1. Update the ACTS CSV (or create a test fixture) with a new column or mapping for the expected outcome.
  2. Implement automated tests in your project using its native testing framework. You may implement them as parameterized/data-driven tests (e.g., JUnit 5 @ParameterizedTest with @CsvFileSource, pytest @pytest.mark.parametrize) or as discrete unit tests mapped to your ACTS combinations and CFG paths.
  3. Execute the test suite locally and verify all tests pass.
  4. Push your branch task/2-modeling to GitLab CSUI, create a Merge Request (MR) targeting your default branch, and ensure the CI/CD pipeline passes.
  5. Verify that your project maintains a passing Quality Gate on SonarQube CSUI with no new blocker or critical quality issues introduced.

Task 4: Generative AI Test Generation Comparison & Reflection

Similar to Exercise 2, evaluate the test suites generated by AI by comparing them against the combinatorial and graph-based models. Prompt a GenAI assistant to generate the unit test suite without prior knowledge of existing test suites in the codebase and without knowledge about this project specifications.

1. Generate Tests via Prompt in Isolation

Copy and paste the following prompt template into a standalone AI chat session on a Web-based chat (e.g., Claude, ChatGPT), or an isolated AI agent session with no access to the source code:

Prompt template for your chosen method:

1
2
3
4
5
You are a senior Software Engineer in Test.
Write a comprehensive suite of unit tests for the following method in <LANGUAGE>.
Focus on boundary values, null/empty inputs, edge cases, and mocking dependencies where necessary.

<PASTE_YOUR_METHOD_CODE_HERE>

2. Compare AI Output Against IDM & CFG

Critically compare the AI-generated test suite against:

  • ACTS PWC Matrix: Did the AI cover all combinations of inputs? Did it test the exact boundary values, or only generic cases?
  • CFG EPC Test Paths: Did the AI exercise all edge pairs, loop iterations, and exception branches?

Document your evaluation and reflection (2–3 paragraphs) in your written report.

Deliverables

At the end of this project, prepare the following artifacts in your fork repository on GitLab CSUI:

  1. Written Report: A Markdown file named PROJECT_2_REPORT.md in the repository root.
  2. ACTS Model & Data Files:
    • The ACTS model file (.xml or .txt) for your chosen method.
    • The exported CSV file generated by ACTS.
    • The augmented CSV / test data file with expected outcomes (stored under your project's test resources or test directory).
  3. CFG Diagram: The exported image file (PNG or JPEG) committed to the repository and referenced in PROJECT_2_REPORT.md.
  4. Codebase Updates:
    • New or updated test files covering your ACTS combinations and CFG paths.
    • All tests passing locally and in CI/CD.
  5. Quality Gate Verification:
    • Passing CI/CD pipeline and passing SonarQube Quality Gate on SonarQube CSUI.

The due date of this project is: Sunday, 22 November 2026, 23:55 UTC+7.

Submit the URL of your repository on GitLab CSUI to the designated submission slot on SCELE before the deadline.

GenAI Usage Policy

This project allows GenAI and LLM tools as learning materials and aids. Even so, it is still imperative for you as a course participant of SQA to critically review and factually verify all outputs generated by GenAI.

The use of GenAI does not mean you can entirely delegate your learning process to GenAI. You enrolled in this course to learn and obtain knowledge and skills. Do not make GenAI the one that does the learning, turning you into a "meat proxy" for it.

Allowed Uses

  • Conducting the comparative evaluation required in Task 4 using the provided prompt in isolation.
  • Asking for documentation and syntax examples regarding your project's test framework (e.g. parameterized test syntax in JUnit/pytest).
  • Brainstorming potential semantic partitions and edge cases.
  • Directing assistants to help debug failing test runs or pipeline errors.

Prohibited Uses

  • Direct, uncritical copy-pasting of AI-generated tables, graphs, or calculations without verification.
  • Outsourcing the construction of the CFG or IDM partitions entirely to an AI tool.

Report Template

Use the following Markdown template for PROJECT_2_REPORT.md:

Note: you can write your report in English or Bahasa Indonesia.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# Project 2 Report: Combinatorial & Model-Based Testing

- **Name:** <NAME>
- **NPM:** <NPM>
- **GitLab Repository URL:** <URL>
- **SonarQube Project URL:** <URL>
- **Programming Language & Test Framework:** <e.g., Python / pytest, , Java / JUnit 5>

## 1. Selected Target Method

- **Method Identifier:** `<e.g., calculate_discount() in billing/service.py>`
- **Method Signature & Context:**
  <Paste the method signature and brief context>
- **Rationale for Selection:**
  `<Explain why this method has non-trivial domain logic, branching, and suitability for IDM and CFG.>`

## 2. Input Domain Modeling with NIST ACTS

### Functionality-Based Characteristics & Partitions

| Characteristic | Description | Partition Blocks |
| :--- | :--- | :--- |
| `<Char 1>` | `<Description>` | `[Block 1, Block 2, ...]` |
| `<Char 2>` | `<Description>` | `[Block 1, Block 2, ...]` |
| `<...>` | `<...>` | `[...]` |

### Constraints

- `<Enumerate constraints formulated in ACTS, or state None if unconstrained>`

### ACTS Pair-Wise Coverage (PWC) Results

- **ACTS Configuration:** Degree of Interaction = 2, Algorithm = IPOG
- **Number of Generated Test Configurations:** `<N>`
- `<Attach or summarize generated table>`

## 3. Control Flow Graph (CFG) & Edge-Pair Coverage (EPC)

- **CFG Diagram:** `![CFG Diagram](path/to/cfg_diagram.png)`
- **Edge-Pair Requirements:** `<List of edge-pairs>`
- **Test Paths:** `<Enumerate test paths and label whether each path is feasible or not>`

## 4. Automated Test Implementation & Quality Gate

- **Test Implementation Strategy:** `<Describe whether you used parameterized/data-driven tests or discrete test methods>`
- **Local Test Execution Output:**
  `<Paste snippet of passing test run from terminal/IDE>`
- **GitLab CI/CD Pipeline Status:** `<Put pipeline badge showing the GitLab CI/CD pipeline status>`
- **SonarQube Quality Gate Status:** `<Put SonarQube CSUI quality gate badge>`

## 5. AI Test Generation Comparison & Reflection

- **AI Tool Used:** `<e.g., ChatGPT / Claude / Copilot in isolated session>`
- **Comparison Against ACTS PWC Matrix:**
  `<Discuss which combinations and boundary values are missed by the AI-generated tests.>`
- **Comparison Against CFG EPC Paths:**
  `<Discuss which control paths, loop edge cases, or exception paths are missed by the AI-generated tests.>`
- **Reflection (2–3 paragraphs):**
  `<Critique of LLM test generation capabilities on your real-world method vs systematic models.>`

## 6. Interesting/Relevant Findings (optional)

> Write down any interesting findings, challenges with real-world codebases, or tool quirks encountered during this project.

GenAI Use Disclosure

Antigravity CLI with Gemini 3.8 Flash model was used to proofread this document and generate the report template. We have reviewed and edited the generated text.

References

  • NIST Automated Combinatorial Testing for Software (ACTS): https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software
  • Paul Ammann & Jeff Offutt, Introduction to Software Testing, 2nd Edition (Cambridge University Press).
  • Chapter 6: Input Space Partitioning
  • Chapter 7: Graph Coverage
Last update: 2026-11-14 01:30:54
Created: 2026-11-14 01:30:54