What is a Test Runner in Cucumber?

A Test Runner is a Java class in a Cucumber project that serves as the entry point for running tests. It uses the @RunWith(Cucumber.class) annotation to integrate with JUnit and the @CucumberOptions annotation to configure test execution, specifying details like feature file locations, step definition packages, tags, and plugins. The Test Runner acts as a coordinator, ensuring that Cucumber executes your scenarios correctly and generates the desired reports.

Why Use a Test Runner?

  • Centralized Configuration: Define all test execution settings in one place.
  • Test Execution: Run feature files and step definitions seamlessly with JUnit.
  • Customization: Filter tests by tags, select report formats, and enable/disable features.
  • CI/CD Integration: Integrate with build tools like Maven and CI systems like Jenkins.
  • Scalability: Manage large test suites with clear organization.

How a Test Runner Works

The Test Runner class is annotated with:

  • @RunWith(Cucumber.class): Tells JUnit to use Cucumber’s test runner instead of the default JUnit runner.
  • @CucumberOptions: Configures options like:
    • features: Path to feature files.
    • glue: Package(s) containing step definitions and hooks.
    • tags: Filters scenarios by tags.
    • plugin: Specifies report formats.
    • monochrome: Improves console output readability.
    • dryRun: Checks for undefined steps without executing tests.
    • strict: Fails tests if undefined or pending steps exist.

When you run the Test Runner class, Cucumber:

  1. Loads the feature files specified in features.
  2. Matches steps to definitions in the glue package(s).
  3. Executes scenarios, respecting tags filters.
  4. Generates reports using the specified plugin options.

Creating and Configuring a Test Runner

Let’s create a feature file, step definitions, and a Test Runner to demonstrate how it orchestrates test execution. Assume you have a Cucumber project set up with Java and Maven, as described in the Installation and Setup post.

Example: Feature File

Create a file named login.feature in src/test/resources/features:

Feature: User Login
  As a user, I want to log in to the application so that I can access my account.

  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

  @regression
  Scenario: Failed login with invalid credentials
    Given the user is on the login page
    When the user enters "user1" and "wrongpass"
    Then the user should see an error message

Step Definitions

Create LoginSteps.java in src/test/java/steps:

package steps;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.When;
import io.cucumber.java.en.Then;

public class LoginSteps {
    @Given("the user is on the login page")
    public void userIsOnLoginPage() {
        System.out.println("Navigating to login page");
    }

    @When("the user enters {string} and {string}")
    public void userEntersCredentials(String username, String password) {
        System.out.println("Entering username: " + username + ", password: " + password);
        if (password.equals("wrongpass")) {
            throw new RuntimeException("Invalid credentials");
        }
    }

    @Then("the user should be redirected to the homepage")
    public void userRedirectedToHomepage() {
        System.out.println("Verifying redirection to homepage");
    }

    @Then("the user should see an error message")
    public void userSeesErrorMessage() {
        System.out.println("Verifying error message");
    }
}

Create a Test Runner

Create TestRunner.java in src/test/java/runner:

package runner;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    tags = "@smoke or @regression",
    plugin = {
        "pretty",
        "html:target/cucumber-reports/cucumber.html",
        "json:target/cucumber-reports/cucumber.json",
        "junit:target/cucumber-reports/cucumber-junit.xml"
    },
    monochrome = true,
    dryRun = false,
    strict = false
)
public class TestRunner {
}

Explanation:

  • features: Points to the directory containing feature files.
  • glue: Specifies the package (steps) containing step definitions.
  • tags: Runs scenarios tagged with @smoke or @regression.
  • plugin: Generates console output (pretty), HTML, JSON, and JUnit XML reports.
  • monochrome: Ensures clean console output.
  • dryRun: Set to false to execute tests (if true, checks for undefined steps only).
  • strict: Set to false to allow undefined/pending steps (if true, fails on undefined steps).

Run the Tests

Run the TestRunner class in your IDE or use Maven:

mvn test

Output:

Feature: User Login
  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

  @regression
  Scenario: Failed login with invalid credentials
    Given the user is on the login page
    When the user enters "user1" and "wrongpass"
      java.lang.RuntimeException: Invalid credentials
      ...
    Then the user should see an error message

2 Scenarios (1 passed, 1 failed)
6 Steps (4 passed, 1 failed, 1 skipped)
0m0.245s

Reports:

  • HTML: target/cucumber-reports/cucumber.html (viewable in a browser).
  • JSON: target/cucumber-reports/cucumber.json (machine-readable).
  • JUnit XML: target/cucumber-reports/cucumber-junit.xml (CI-compatible).

Advanced Test Runner Configurations

Let’s explore advanced configurations to customize test execution.

Example 1: Running Specific Tags

Update TestRunner.java to run only @smoke scenarios:

@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    tags = "@smoke",
    plugin = {"pretty", "html:target/cucumber-reports/smoke.html"},
    monochrome = true
)

Run the tests:

mvn test

Output:

Feature: User Login
  @smoke
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When the user enters "user1" and "pass123"
    Then the user should be redirected to the homepage

1 Scenarios (1 passed)
3 Steps (3 passed)
0m0.123s

The HTML report (target/cucumber-reports/smoke.html) will only include the @smoke scenario.

Example 2: Dry Run Mode

Set dryRun = true to check for undefined steps without executing tests:

@CucumberOptions(
    features = "src/test/resources/features",
    glue = "steps",
    tags = "@smoke or @regression",
    plugin = {"pretty"},
    monochrome = true,
    dryRun = true
)

Run the tests:

mvn test

Output:
Shows steps without executing them, highlighting any undefined steps (none in this case since all steps are defined).

Example 3: Multiple Glue Packages

If step definitions and hooks are in different packages, specify multiple glue paths:

@CucumberOptions(
    features = "src/test/resources/features",
    glue = {"steps", "hooks"},
    tags = "@smoke or @regression",
    plugin = {"pretty", "html:target/cucumber-reports/cucumber.html"},
    monochrome = true
)

This is useful for projects with separate packages for steps and hooks.

Using Test Runner with CLI

You can bypass the Test Runner and run tests directly via the Cucumber CLI, specifying options similar to @CucumberOptions:

mvn cucumber:test -Dcucumber.features=src/test/resources/features -Dcucumber.glue=steps -Dcucumber.filter.tags="@smoke or @regression" -Dcucumber.plugin=pretty,html:target/cucumber-reports/cucumber.html

However, the Test Runner is preferred for:

  • Consistent configuration across team members.
  • IDE integration (e.g., running tests directly in IntelliJ).
  • CI/CD pipelines using Maven or Gradle.

Integrating Test Runner with CI/CD

To integrate the Test Runner with CI/CD tools like Jenkins:

  1. Ensure pom.xml includes the Cucumber and JUnit dependencies.
  2. Configure the Test Runner with a JUnit XML plugin:
    plugin = {"junit:target/cucumber-reports/cucumber-junit.xml"}
  3. In Jenkins, add a build step to run mvn test and a post-build step to publish JUnit reports (target/cucumber-reports/cucumber-junit.xml).
  4. Archive HTML reports as artifacts:
    plugin = {"html:target/cucumber-reports/cucumber.html"}

This allows you to view test results and reports in Jenkins.

Best Practices for Test Runner

  1. Centralize Configuration: Keep all test settings in the Test Runner for consistency.
  2. Use Meaningful Tags: Filter tests with tags (e.g., @smoke, @regression) for targeted execution.
  3. Organize Report Files: Save reports in a dedicated directory (e.g., target/cucumber-reports).
  4. Enable Monochrome: Use monochrome = true for clean CI logs.
  5. Test Dry Run: Periodically use dryRun = true to check for undefined steps.
  6. Keep It Simple: Avoid overly complex glue or plugin configurations.

Troubleshooting Test Runner Issues

  • Feature Files Not Found: Verify the features path is correct (e.g., src/test/resources/features).
  • Step Definitions Not Found: Ensure the glue package matches the step definition location (e.g., steps).
  • Tag Filters Incorrect: Check tag syntax (e.g., @smoke vs. smoke) and logical operators (and, or).
  • Reports Not Generated: Confirm plugin paths are writable and correctly formatted.
  • Test Runner Not Executing: Ensure @RunWith(Cucumber.class) and @CucumberOptions are properly annotated.

Tips for Beginners

  • Start with Basic Configuration: Use minimal features, glue, and plugin options initially.
  • Run from IDE: Test the Test Runner in your IDE (e.g., IntelliJ) for quick feedback.
  • Check Console Output: Use the pretty plugin to verify test execution.
  • Experiment with Tags: Try different tag filters to understand selective execution.