prewarm(promptPrefix:)

Loads the resources required for this session into memory ahead of a request.

Declaration

final func prewarm(promptPrefix: Prompt? = nil)

Mentioned in

Optimizing key-value caching in language model sessions

Discussion

This method can be useful in cases where you have a strong signal that the user will interact with session within a few seconds. For example, you might call prewarm(promptPrefix:) when a person begins typing into a text field.

If you know a prefix for the future prompt, passing it to prewarm(promptPrefix:) allows the system to process the prompt eagerly and reduce latency for the future request.

Calling this method doesn’t guarantee that the system loads your assets immediately, particularly if your app is running in the background or the system is under load.