EvaluatorProtocol
A type that evaluates subjects and produces metrics.
Declaration
protocol EvaluatorProtocol<Input, Subject> : SendableOverview
Conform to EvaluatorProtocol to create custom evaluators that measure the system’s output against expected criteria. Each evaluator returns an array of Metric values: one for each DataFrame column.
The protocol takes Input (the sample type) as a type parameter. Subject is an associated type constrained to EvaluationSubject, ensuring the subject’s value type matches the sample’s expected value type.
Conforming types must be Sendable.
struct MyEvaluator<Input: SampleProtocol>: EvaluatorProtocol
where Input.ExpectedValue: Sendable & Codable {
let metric = Metric("Quality")
func metrics(
subject: ModelSubject<Input.ExpectedValue>,
input: Input
) async throws -> [Metric] {
return [metric.scoring(1.0)]
}
}