Contents

SiriScenarioLoader

A loader that supplies the scenarios an evaluation runs, in the order you write them.

Declaration

struct SiriScenarioLoader

Overview

A scenario loader is the dataset of a Siri evaluation. It holds the scenarios you pass in, and the evaluation runs each one through Siri in that order. The loader’s closure accepts individual scenarios or arrays of scenarios, so you can combine scenarios you define separately with ones you build from a collection.

The loader holds scenarios but doesn’t set up your app. Set up any app state in your test before the evaluation runs, and group scenarios that start from the same state in one loader. Scenarios that need a different starting state belong in their own evaluation and test because setup runs once per test, not between scenarios.

This example combines separate scenarios in a loader:

let quizExampleIntents = IntentDefinitions(bundleIdentifier: "com.example.QuizExample")
let startQuizIntent = quizExampleIntents.intents["StartQuizIntent"]
let openQuizIntent = quizExampleIntents.intents["OpenQuizIntent"]
let quizTopicEntity = quizExampleIntents.entities["QuizTopicEntity"]

// A scenario you define separately.
let openScienceQuiz = SiriScenario {
    UserTurn("Open the Science quiz in QuizExample") {
        Calls {
            openQuizIntent
                .expect(target: quizTopicEntity.expectIdentifier("Science"))
        }
    }
}

let topics = ["Science", "History", "Geography"]

let dataset = SiriScenarioLoader {
    // Include the scenario you defined separately.
    openScienceQuiz

    // Add one scenario for each topic.
    topics.map { topic in
        SiriScenario {
            UserTurn("In QuizExample, start a \(topic.lowercased()) quiz") {
                Calls {
                    startQuizIntent
                        .expect(topic: .equals(topic))
                }
            }
        }
    }
}
    
// Pass the loader to your custom evaluation type.
let evaluation = QuizExampleEvaluation(name: "Quiz requests", dataset: dataset)

Topics

Creating a loader

Loading scenarios

See Also

Scenarios