0024-PER-TEST-CASE-REPETITIONS: Test case repetition

Introduction

Since its initial release, Swift Testing has supported repeating tests for a number of iterations, or until a specific failure/success condition is reached. This is not currently exposed through swift test, but we would like to make it available.

Currently, when a repetition condition is met by any of the test cases in the test target, all of the tests in the test target are repeated. This proposal seeks to change this iteration behavior to apply the repetition behavior to only those test cases that met the repetition condition.

Motivation

Test repetition is a useful tool for debugging flaky tests or spurious failures, or dealing with unreliable external dependencies.

The current behavior causes unnecessary additional test execution; if only one test case of a large parameterized test suite fails, it will cause all of the tests in the test target to run again. This also does not match the behavior of XCTest, which repeats failing test methods instead of all tests within an XCTestCase or the entire test target.

It also does not match developer expectations; the phrase "repeat tests while issue recorded" implies that only tests which have issues recorded will be repeated, but that is not the current behavior.

Proposed solution

Swift Testing behavior

First, change the repetition behavior for existing harnesses. Swift Testing will only re-run test cases that meet the repetition condition. Further, iterations should be reported directly along with testStarted/testEnded events, not via global iteration events.

swift test command

Add two command-line flags to swift test to support this new behavior, --maximum-repetitions and --repeat-until.

Detailed design

JSON schema changes

Add a per-test-case iteration value to event records in the 6.4 schema version. This value is one-indexed and will be provided even if the configuration has set the repetition policy to .none.

diff --git a/Documentation/ABI/JSON.md b/Documentation/ABI/JSON.md
index f4ae1b84..f315244b 100644
--- a/Documentation/ABI/JSON.md
+++ b/Documentation/ABI/JSON.md
@@ -205,6 +205,7 @@ sufficient information to display the event in a human-readable format.
   ["attachment": <attachment>,] ; the attachment (if kind is "valueAttached")
   "messages": <array:message>,
   ["testID": <test-id>,]
+  ["iteration": <number>] ; the one-indexed test iteration (if event is posted during test execution).
 }

The Tools SPI types will be updated to provide this information as well.

Behavior changes

Individual test cases will have repetition conditions evaluated after every execution and console output will be updated to match. The global iteration behavior will be fully removed.

This does change observable behavior; if a serialized test suite expected a certain set of tests to be run in order, running some of them multiple times without running others may break existing behavior. Such reliance, however, is an anti-pattern, and hidden dependencies like these between test functions should be avoided.

swift test flags

The following flags will be added:

| Flag | Description | | ======================= | =========== | | --maximum-repetitions | The maximum number of times a test should be repeated. | | --repeat-until | The condition by which a test should be repeated. This accepts either pass or fail, and if it is not provided, repetition is unconditional (and all tests will repeat maximum-iterations times) |

Source compatibility

This is purely additive with regards to the JSON schema. Clients of existing Tools SPI will need to be updated to use the updated SPI values.

Integration with supporting tools

If tools intend to support test repetition, they can provide the repetition behavior by providing the existing --repetitions and --repeat-until command-line arguments in the Swift Testing entrypoint. If not provided, all iteration values provided in JSON events will be 1. Note that swift test will accept --maximum-repetitions, but the existing Swift Testing flag is --repetitions. We believe maximum-repetitions to be a clearer name for user-facing flags.

Future directions

Repetition trait

Developers may want to define repetition behaviors for specific tests that override the global value passed in via configuration parameters. Such a trait could be useful for repeating known-flaky tests in CI, for example.

@Test(.repeating(.whileIssueRecorded, maximumIterations: 5, comment: "This might get transiently disconnected")))
func somethingNetworkBound() {
    let value = await downloadSomethingFromTheInternet()
}

More consideration needs to be done with how this interacts with the existing top-level configuration property, but this has been requested in the past.

This also doesn't reach the full level of granularity that the behavior would support; we would need some other syntax for repeating just a subset of parameterized test cases.

@Test(
    .repeating(.whileIssueRecorded, maximumIterations: 5, "This might get transiently disconnected")),
    arguments: [
        TestEnvironment.production,
        .staging, // How can I specify that we should only repeat the test on `.staging`?
        .development
    ]
)
func somethingNetworkBound(env: TestEnvironment) {
    let value = await downloadSomethingFromTheInternet(environment: env)
    #expect(...)
}

Exposing current iteration at runtime

We could add an accessor for clients to read the current iteration at runtime, which would enable developers to do things like adding additional more expensive logging when a test retries to aid in debugging the failure, without incurring that cost for most of their test runs.

@Test
func ableToConnectToSocket() {
    let value = await socket.connect(enableVerboseLogging: Test.currentIteration > 1)
    #expect(...)
}

Alternatives considered

Configuration for global vs case-level iteration

We could provide a toggle in Configuration for choosing the existing global iteration behavior. However, in our experience and discussion with Swift Testing clients, the current behavior is almost always seen as surprising and unexpected. As such, we see little value in leaving the old behavior in beyond the amount required for staging a transition.

Acknowledgments

Thanks to @grynspan and @stmontgomery for helping me iterate this proposal and get the implementation solid, and to the Testing Workgroup for the discussion that led to this proposal.