Evaluations

Overview

Use the Evaluations framework to systematically evaluate your app’s intelligence-powered features. Define datasets, generate model responses, apply metrics, and aggregate results, all with type-safe Swift APIs that integrate directly into your development workflow.

[Image]

With the Evaluations framework, you can:

• Compare prompt strategies.

• Track quality over time.

• Catch regressions before they ship.

The framework evaluates your intelligence-powered features against the metrics you define, from simple pass or fail checks to detailed scoring with model-judge patterns. It aggregates the results into summaries that show you which approach performs best and where individual responses fall short. The framework works with any model available through Foundation Models, including on-device, Private Cloud Compute, and other models.

What’s new

• Meet the Evaluations framework
• Improve your prompts by hill-climbing with Evaluations
• Create robust evaluations for agentic apps

Topics

Essentials

• Evaluating language model responses
• Designing effective evaluations
• Book Tracker: Using Evaluations to evaluate an intelligent feature
• Evaluating a Core AI model
• Evaluation

Datasets

• Generating synthetic datasets
• Designing datasets to test your feature
• ModelSample
• Loader
• SampleGenerator

Metrics and evaluators

• Designing specific, measurable criteria in an evaluation suite
• Metric
• Evaluator
• MetricsAggregator

Results

• EvaluationResult
• ResultColumn
• inputColumn
• responseColumn
• expectedColumn

Model-judge evaluations

• Designing effective model-judge evaluators
• Scoring with model-judge evaluators
• ModelJudgeEvaluator
• ModelJudgePrompt
• ScoreDimension

Tool-call evaluation

• Evaluating tool-calling behavior
• ToolCallEvaluator
• TrajectoryExpectation
• ArgumentMatcher

Swift Testing integration

• EvaluationTrait
• EvaluationContext

Structures

• EvaluationRunErrors