Machine Learning & AI Group Lab

Transcript

Engineers from the machine learning and AI frameworks teams introduce themselves and set up a session covering the Foundation Models framework, on-device versus Private Cloud Compute models, Core ML and Core AI, MLX, bringing custom and local models to Xcode, and evaluation-driven development.

Drive the decision with evaluations. Concretely, the main differences are context size and capability: the on-device system model has a 4K context, while the Private Cloud Compute server model offers 32K and stronger reasoning. Use on-device for lightweight, latency- and privacy-sensitive tasks that fit its window, and reach for PCC when you need a larger context or deeper reasoning.

The stray end-of-turn token points at the chat template — you're on the right track. This is a model-serving detail: ensure the chat template you configure in your MLX server matches the model (e.g. Qwen), since a mismatched template leaves the end token in the output. It's not an Xcode/ACP issue but a local-model configuration one.

Yes, with a correction: Core AI is purely an inference framework, not a training one. To get agentic project manipulation (beyond chat), integrate your local model into Xcode over the ACP protocol rather than the chat path. You can bring your own model served via MLX; the ACP integration is what gives the agent the tools to act on the project.

The practical signal today is the OS version. The on-device model ships as part of the OS and only changes with an OS update — it isn't pushed silently underneath a fixed OS — so key your evaluation gating to OS version. Re-run your evaluation suite against new OS seeds to catch prompt regressions before release.

Yes — the on-device model can run in the background and in widgets. But under system load you may hit rate limiting, which simply means back off and try again later. Design around this: treat classification as best-effort with retry, rather than assuming an immediate background result when the device is busy or under thermal pressure.

Match the tool to the goal. Foundation Models can't output images, so for stylized profile images bring an image-generation model — one option is to prompt the on-device model with the user's app data to build a per-user description, then feed that to your image model. For text-based adaptive guidance, Foundation Models fits well, often via prompt engineering and in-context learning rather than full fine-tuning. If you do want to train or fine-tune your own model, MLX is a good starting point and you can run the result on device with Core AI; per-user adapters are a lighter option, though not for diffusion models. Evaluations can guide the process — progressively improving quality and even synthesizing test data with its sample generator.

It's a language model, so it excels at content extraction and generation — and with this year's image input, many multimodal tasks too. The clear limit is that it can't produce images as output (it's not a diffusion model), so don't reach for it for image generation; use a diffusion/image model for that instead.

Swift Playgrounds is a friendly on-ramp — its intro-to-ML content builds a Core ML image classifier (a rock-paper-scissors game using hand gestures) and teaches how to assemble a diverse dataset and train a model end to end. From there, move on to the Core ML tools and the WWDC ML sessions.

Generally, if you can express your model in PyTorch and it's exportable through PyTorch's export APIs, it converts to Core AI straightforwardly. The exceptions are newer operations and techniques that aren't yet exportable — when a model won't export, inspect the PyTorch export APIs to see which operations are unsupported and adapt accordingly. MLX is also a good place to experiment with PyTorch-style models.

Check the model's availability property early and design a graceful fallback for when it's unavailable, rather than assuming it's present. For latency, use the prewarm API on the on-device system language model so the first request isn't cold. Together, availability checks plus prewarming give a responsive experience that degrades cleanly on unsupported devices.

You don't have to abandon Core ML. Core ML remains supported and available on current devices, while Core AI is the newer path; you can adopt Core AI where it helps and keep Core ML for broad device coverage. Plan the migration around your deployment targets rather than waiting on a single cutover.

There's no single fixed number — it depends on the device and available memory. Check the OS process available-memory APIs to see how much you can safely use, and size your bundled model to fit within that budget with headroom. When a model is too large to run reliably on target devices, that's the signal to move the workload to Private Cloud Compute.

Start from the fundamental experience you want to deliver, not from adding AI for its own sake. Like any framework in the ecosystem, use these tools only where they genuinely improve the experience for a specific need — restraint is part of good design, and AI should recede when it doesn't add clear value.

Yes — this year's LanguageModel protocol lets you drive any model through the same unified Swift API for prompting, tool calling, and @Generable output, so a custom provider around your server backend can serve a single LanguageModelSession across all devices without rewriting how you prompt. Start by checking whether your backend conforms to the typical OpenAI chat-completions request shape; if it does, this year's open-source utilities package already includes a conforming implementation you can use as-is. You can also mix models in one app — deciding which to use can be as simple as an if statement based on platform availability, while the rest of your code stays the same.