0013-ISSUE-SEVERITY-WARNING: Test Issue Severity

Introduction

I propose introducing a new API to Swift Testing that allows developers to record issues with a specified severity level. By default, all issues will have severity level “error”, and a new “warning” level will be added to represent less severe issues. The effects of the warning recorded on a test will not cause a failure but will be included in the test results for inspection after the run is complete.

Motivation

Currently, when an issue arises during a test, the only possible outcome is to mark the test as failed. This presents a challenge for users who want a deeper insight into the events occurring within their tests. By introducing a dedicated mechanism to record issues that do not cause test failure, users can more effectively inspect and diagnose problems at runtime and review results afterward. This enhancement provides greater flexibility and clarity in test reporting, ultimately improving the debugging and analysis process.

Use Cases

• Warning about a Percentage Discrepancy in Image Comparison:

- Scenario: When comparing two images to assess their similarity, a warning can be triggered if there's a 95% pixel match, while a test failure is set at a 90% similarity threshold. - Reason: In practices like snapshot testing, minor changes (such as a timestamp) might cause a discrepancy. Setting a 90% match as a pass ensures test integrity. However, a warning at 95% alerts testers that, although the images aren't identical, the test has passed, which may warrant further investigation.

• Warning for Duplicate Argument Inputs in Tests:

- Scenario: In a test library, issue a warning if a user inputs the same argument twice, rather than flagging an error. - Reason: Although passing the same argument twice might not be typical, some users may have valid reasons for doing so. Thus, a warning suffices, allowing flexibility without compromising the test's execution.

• Warning for Recoverable Unexpected Events:

- Scenario: During an integration test where data is retrieved from a server, a warning can be issued if the primary server is down, prompting a switch to an alternative server. Usually mocking is the solution for this but may not test everything needed for an integration test. - Reason: Since server downtime might happen and can be beyond the tester's control, issuing a warning rather than a failure helps in debugging and understanding potential issues without impacting the test's overall success.

• Warning for a retry during setup for a test:

- Scenario: During test setup part of your code may be configured to retry, it would be nice to notify in the results that a retry happened - Reason: This makes sense to be a warning and not a failure because if the retry succeeds the test may still verify the code correctly

Proposed solution

We propose introducing a new property on Issue in Swift Testing called severity, that represents if an issue is a warning or an error. The default Issue severity will still be error and users can set the severity when they record an issue.

Test authors will be able to inspect if the issue is a failing issue and will be able to check the severity.

Detailed design

Severity Enum

We introduce a Severity enum to categorize issues detected during testing. This enum is crucial for distinguishing between different levels of test issues and is defined as follows:

The Severity enum:

extension Issue {
  // ...

  public enum Severity: Codable, Comparable, CustomStringConvertible, Sendable {
    /// The severity level for an issue which should be noted but is not
    /// necessarily an error.
    ///
    /// An issue with warning severity does not cause the test it's associated
    /// with to be marked as a failure, but is noted in the results.
    case warning

    /// The severity level for an issue which represents an error in a test.
    ///
    /// An issue with error severity causes the test it's associated with to be
    /// marked as a failure.
    case error
  }
}

Recording Non-Failing Issues

To enable test authors to log non-failing issues without affecting test results, we provide a method for recording such issues:

Issue.record("My comment", severity: .warning)

Here is the Issue.record method definition with severity as a parameter.

extension Issue {
  // ...

  /// Record an issue when a running test and an issue occurs.
  ///
  /// - Parameters:
  ///   - comment: A comment describing the expectation.
  ///   - severity: The severity level of the issue.  This factor impacts whether the issue constitutes a failure.  
  ///   - sourceLocation: The source location to which the issue should be
  ///     attributed.
  ///
  /// - Returns: The issue that was recorded.
  ///
  /// Use this function if, while running a test, an issue occurs that cannot be
  /// represented as an expectation (using the ``expect(_:_:sourceLocation:)``
  /// or ``require(_:_:sourceLocation:)-5l63q`` macros.)
  @discardableResult public static func record(
    _ comment: Comment? = nil,
    severity: Severity = .error,
    sourceLocation: SourceLocation = #_sourceLocation
  ) -> Self
}

Issue Type Enhancements

The Issue type is enhanced with two new properties to better handle and report issues:

• severity: This property allows access to the specific severity level of an issue, enabling more precise handling of test results.

extension Issue {
  // ...

  /// The severity of the issue.
  public var severity: Severity { get set }
}

• isFailure: A boolean computed property to determine if an issue results in a test failure, thereby helping in result aggregation and reporting.

extension Issue {
  // ...

  /// Whether or not this issue should cause the test it's associated with to be
  /// considered a failure.
  ///
  /// The value of this property is `true` for issues which have a severity level of
  /// ``Issue/Severity/error`` or greater and are not known issues via
  /// ``withKnownIssue(_:isIntermittent:sourceLocation:_:when:matching:)``.
  /// Otherwise, the value of this property is `false.`
  ///
  /// Use this property to determine if an issue should be considered a failure, instead of
  /// directly comparing the value of the ``severity`` property.
  public var isFailure: Bool { get }
}

Example usage of severity and isFailure:

withKnownIssue {
  // ...
} matching: { issue in
  issue.isFailure || issue.severity > .warning
}

For more details on Issue, refer to the Issue Documentation.

This revision aims to clarify the functionality and usage of the Severity enum and Issue properties while maintaining consistency with the existing Swift API standards.

Source compatibility

The aspect of this proposal which adds a new severity: parameter to the Issue.record function introduces the possibility of a source breakage for any clients who are capturing a reference to the function. Existing code could break despite the fact that the new parameter specifies a default value of .error. Here's a contrived example:

// ❌ Source breakage due to new `Issue.Severity` parameter
let myRecordFunc: (Comment?, SourceLocation) -> Issue = Issue.record

To avoid source breakage, we will maintain the existing overload and preserve its signature, but mark it deprecated, disfavored, and hidden from documentation:

extension Issue {
  // ...

  @available(*, deprecated, message: "Use record(_:severity:sourceLocation:) instead.")
  @_disfavoredOverload
  @_documentation(visibility: private)
  @discardableResult public static func record(
    _ comment: Comment? = nil,
    sourceLocation: SourceLocation = #_sourceLocation
  ) -> Self
}

Integration with supporting tools

Event stream

Issue severity will be in the event stream output when a issueRecorded event occurs. This will be a breaking change because some tools may assume that all issueRecorded events are failing. Due to this we will be bumping the event stream version and v1 will maintain it's behavior and not output any events for non failing issues. We will also be adding isFailure to the issue so that clients will know if the issue should be treated as a failure. isFailure is a computed property.

The JSON event stream ABI will be amended correspondingly:

 <issue> ::= {
   "isKnown": <bool>, ; is this a known issue or not?
+  "severity": <string>, ; the severity of the issue
+  "isFailure": <bool>, ; if the issue is a failing issue
   ["sourceLocation": <source-location>,] ; where the issue occurred, if known
 }

Example of an issueRecorded event in the json output:

{"kind":"event","payload":{"instant":{"absolute":302928.100968,"since1970":1751305230.364087},"issue":{"_backtrace":[{"address":4437724864},{"address":4427566652},{"address":4437724280},{"address":4438635916},{"address":4438635660},{"address":4440823880},{"address":4437933556},{"address":4438865080},{"address":4438884348},{"address":11151272236},{"address":4438862360},{"address":4438940324},{"address":4437817340},{"address":4438134208},{"address":4438132164},{"address":4438635048},{"address":4440836660},{"address":4440835536},{"address":4440834989},{"address":4438937653},{"address":4438963225},{"address":4438895773},{"address":4438896161},{"address":4438891517},{"address":4438937117},{"address":4438962637},{"address":4439236617},{"address":4438936181},{"address":4438962165},{"address":4438639149},{"address":4438935045},{"address":4438935513},{"address":11151270653},{"address":11151269797},{"address":4438738225},{"address":4438872065},{"address":4438933417},{"address":4438930265},{"address":4438930849},{"address":4438909741},{"address":4438965489},{"address":11151508333}],"_severity":"error","isFailure":true, "isKnown":false,"sourceLocation":{"_filePath":"\/Users\/swift-testing\/Tests\/TestingTests\/EntryPointTests.swift","column":23,"fileID":"TestingTests\/EntryPointTests.swift","line":46}},"kind":"issueRecorded","messages":[{"symbol":"fail","text":"Issue recorded"},{"symbol":"details","text":"Unexpected issue Issue recorded (warning) was recorded."}],"testID":"TestingTests.EntryPointTests\/warningIssues()\/EntryPointTests.swift:33:4"},"version":0}

Console output

When there is an issue recorded with severity warning, such as using the following code:

Issue.record("My comment", severity: .warning)

the console output will look like the following:

◇ Test "All elements of two ranges are equal" started.
� Test "All elements of two ranges are equal" recorded a warning at ZipTests.swift:32:17: Issue recorded
↳ My comment
✔ Test "All elements of two ranges are equal" passed after 0.001 seconds with 1 warning.

Alternatives considered

• Separate Issue Creation and Recording: We considered providing a mechanism to create issues independently before recording them, rather than passing the issue details directly to the record method. This approach was ultimately set aside in favor of simplicity and directness in usage.

• Naming of isFailure vs. isFailing: We evaluated whether to name the property isFailing instead of isFailure. The decision to use isFailure was made to adhere to naming conventions and ensure clarity and consistency within the API.

• Severity-Only Checking: We deliberated not exposing isFailure and relying solely on severity checks. However, this was rejected because it would require test authors to overhaul their code should we introduce additional severity levels in the future. By providing isFailure, we offer a straightforward way to determine test outcome impact, complementing the severity feature.
• Naming Severity.error Severity.failure instead because this will always be a failing issue and test authors often think of test failures. Error and warning match build naming conventions and XCTest severity naming convention.

Future directions

• In the future I could see the warnings being able to be promoted to errors in order to run with a more strict testing configuration

• In the future I could see adding other levels of severity such as Info and Debug for users to create issues with other information.

Acknowledgments

Thanks to Stuart Montgomery for creating and implementing severity in Swift Testing.

Thanks to Joel Middendorf, Dorothy Fu, Brian Croom, and Jonathan Grynspan for feedback on severity along the way.