InferenceFunction.AsyncValue
A future which will provide an inference value once any pending write is complete.
Declaration
final class AsyncValueOverview
An AsyncValue contains an underlying InferenceValue however that value may be actively in-use by some previously dispatched async work, and thus accessing the underlying value below an AsyncValue requires an await to wait for any previous compute writing it to be complete.
An AsyncValue is immutable once any previous compute has completed.
Async values can be used in async pipelines of inference to dispatch multiple inference functions in sequence without waiting for each to complete before dispatching the next. This can improve performance by parallelizing phases of the inferences which are not data dependent:
// Pipeline encoding of a text embedding function followed by decoder
var textTokens: NDArray = ...
let embeddingOutputs = try textEmbeddingFunction.encode(inputs: ["tokens": .init(textTokens)])
let embeddings: InferenceFunction.AsyncValue = embeddingsOutputs["embeddings"]
let decoderOutputs = try decodingFunction.encode(inputs: ["embeddings": embeddings])
let logits = decoderOutputs["logits"]!
// Await the compute of logits to be complete
let logitsNDArray = try await logits.ndArray