Contents

Siri Evaluations

Evaluate how well Siri engages your app’s intents and entities by running scripted conversations and checking what Siri does.

Overview

Use the Siri Evaluations framework to script requests to Siri and measure how Siri handles them, with metrics you can check in a test. The test records which intents Siri calls and which entities Siri finds, and compares each step against your expectations. Use it to catch changes in how Siri responds as you update your intents and entities.

[Image]

Write a conversation as a scenario that contains user turns. In each turn, you describe what the person asks and what you expect Siri to do. Your expectations can include the intents Siri calls, the parameter values Siri passes, the entities Siri finds, and the text Siri responds with. Each check compares what Siri does with the values you specify, and the comparison is deterministic.

Start with a few scenarios that use simple, direct requests to confirm that your integration works end to end, alongside trying requests with Siri on a device. Then build a broad set of scenarios that reflects the many ways people phrase requests, and include indirect, incomplete, and unexpected wording. Because Siri’s responses can vary between runs, run the set more than once, and improve your intents, their parameter names and descriptions, and your App Skills until more of the set passes, and passes consistently.

Siri Evaluations works with the Evaluations and Swift Testing frameworks. If you’ve used either one, the workflow is similar to what you do in these frameworks:

  • Define a Siri evaluation.

  • Run it from a test.

  • Check the metrics it produces.

After a run, Xcode shows a report of what Siri says and does in each turn.

Topics

Evaluations

Scenarios

Expectations

Results