Rendering Gaussian splats with RealityKit

Overview

A Gaussian splat asset represents a scene as a large collection of colored, oriented ellipsoids instead of a mesh, so a real-world capture keeps details that are hard to model by hand, like soil texture or the fine edges of leaves. Each splat carries position, scale, rotation, opacity, and a set of spherical harmonic coefficients that let its color shift depending on the angle you view it from. This reproduces specular highlights and reflections that were present in the original capture. The capture bakes that color in, so scene lighting doesn’t change how a splat looks, and a splat won’t cast a shadow the way mesh-based content does.

[Image]

This sample uses RealityKit to bring a scan of a potted plant into a mixed immersive space. The app loads the same plant from either a USD file or a PLY file, then lets the person wearing the device pick the plant up, move it, and anchor it to a nearby table or floor. A scene can only render a limited number of splat entities at once, so budget them the way you would any other GPU-heavy resource.

Configure the sample code project

Because Gaussian splats don’t render in the Simulator, ensure your splats render properly on device.

Load a splat from a USD file

The sample’s bundled USD asset stores its splat data in a primitive whose schema type is ParticleField3DGaussianSplat. This primitive isn’t defined by RealityKit or USDKit, and is instead created by the tool that authored the asset. loadUSDEntity(assetName:) opens the stage with USDStage, finds the primitive with that schema using isSchema(_:), and reads its attributes into a GaussianSplatBuffers:

func loadUSDEntity(assetName: String) async throws -> Entity {
    let url = try bundleUSDURL(assetName: assetName)

    let buffers = try await Task.detached(priority: .userInitiated) {
        let stage = try USDStage.open(url)

        guard let splatPrim = stage.descendants.first(where: {
            $0.isSchema("ParticleField3DGaussianSplat")
        }) else {
            throw GaussianSplatError.invalidData("No ParticleField3DGaussianSplat prim found in \(url.lastPathComponent)")
        }

        return try buildBuffers(from: splatPrim)
    }.value

    return try makeSplatEntity(from: buffers, isLinear: true)
}

RealityKit expects each rotation as four contiguous floats in w, x, y, z order. USDValue.Quatf exposes its parts through the real and imaginary accessors rather than as raw fields in that layout, so the loop reads each component through its accessor and writes it into the matching slot instead of copying the struct’s bytes directly:

private nonisolated func fillRotationBuffer(_ buffer: LowLevelBuffer?, quatArray: [USDValue.Quatf], count: Int) {
    buffer?.withUnsafeMutableBytes { dst in
        let out = dst.bindMemory(to: Float.self)
        for idx in 0..<count {
            let quat = quatArray[idx]
            out[idx * 4 + 0] = quat.real          // w
            out[idx * 4 + 1] = quat.imaginary.x   // x
            out[idx * 4 + 2] = quat.imaginary.y   // y
            out[idx * 4 + 3] = quat.imaginary.z   // z
        }
    }
}

For more information on providing raw splat data for rendering Gaussian splats in RealityKit, see GaussianSplatComponent.

Load a splat from a PLY file

PLY is a generic 3D file format and its header declares the elements and properties inside the file. PLY doesn’t have a standard schema for Gaussian splat data. This sample expects a plain-text header that declares each property, followed by a binary block with one record per splat, using the property set that Gaussian splatting tools have settled on as a common convention:

loadPLYEntity(assetName:) reads the file, deinterleaves those properties into a GaussianSplatBuffers, and calls makeSplatEntity(from:) without isLinear, so RealityKit converts the log-space scale and unbounded opacity:

func loadPLYEntity(assetName: String) async throws -> Entity {
    let url = try bundlePLYURL(assetName: assetName)
    let buffers = try await Task.detached(priority: .userInitiated) {
        let splatData = try readGaussianSplatFile(url)
        return try deinterleaveGaussianSplatData(splatData)
    }.value
    return try makeSplatEntity(from: buffers)
}

The deinterleaveGaussianSplatData(_:) method also sanitizes every value it copies, replacing any NaN or infinite float with 0. RealityKit rejects a splat buffer that contains a non-finite value, so a loader reading arbitrary capture data needs to clean it up first.

Prepare buffers for splat assets

Whichever format the sample starts from, it fills the same GPU buffers for position, scale, rotation, opacity, and spherical harmonics. This sample defines a custom structure, GaussianSplatBuffers, that holds those buffers, and a helper turns a filled-in structure into something the app can add to a scene. assembleSplatComponent(from:isLinear:) packages the buffers into a GaussianSplatResource.BufferResource, wraps that in a GaussianSplatResource, and builds a GaussianSplatComponent from it. assembleSplatComponent(from:isLinear:) also sets the resource’s scaleActivation and opacityActivation, which tell RealityKit how to interpret the raw numbers in the buffers:

func assembleSplatComponent(from buffers: GaussianSplatBuffers, isLinear: Bool = false) throws -> GaussianSplatComponent {
    guard let pos = buffers.positionBuffer,
          let scale = buffers.scaleBuffer,
          let rotation = buffers.rotationBuffer,
          let opacity = buffers.opacityBuffer,
          let shBuf = buffers.shBuffer else {
        throw GaussianSplatError.invalidData("One or more GPU buffers failed to allocate")
    }

    let degree = GaussianSplatResource.SphericalHarmonicDegree(rawValue: buffers.degreeSH) ?? .zero
    let bufferResource = try GaussianSplatResource.BufferResource(
        count: Int(buffers.splatCount),
        position: makeDescriptor(buffer: pos, format: .float3, stride: 3 * 4),
        scale: makeDescriptor(buffer: scale, format: .float3, stride: 3 * 4),
        rotation: makeDescriptor(buffer: rotation, format: .float4, stride: 4 * 4),
        opacity: makeDescriptor(buffer: opacity, format: .float, stride: 1 * 4),
        sphericalHarmonics: (makeSHDescriptor(buffer: shBuf, tupleSH: buffers.tupleSH), degree)
    )

    let splatResource = GaussianSplatResource(bufferResource)
    if isLinear {
        splatResource.scaleActivation   = .identity
        splatResource.opacityActivation = .identity
    } else {
        splatResource.scaleActivation   = .exponential
        splatResource.opacityActivation = .sigmoid
    }
    return GaussianSplatComponent(splatResource)
}

This sample passes isLinear: true for data that’s already in linear space, like the USD version of the plant asset. For the PLY asset, the sample leaves it false, allowing RealityKit to apply the .exponential and .sigmoid conversions for splat data.

Add interactivity to a splat entity

A splat entity is an Entity like any other, so it takes the same components. The sample adds a GroundingShadowComponent and a ManipulationComponent so a person can pick up the plant and move it.

private func configureManipulableObject(_ entity: Entity) {
    ManipulationComponent.configureEntity(
        entity,
        collisionShapes: [ShapeResource.generateBox(size: SIMD3<Float>(repeating: 0.5))]
    )
    var manipulation = entity.components[ManipulationComponent.self] ?? ManipulationComponent()

    // ...

    entity.components.set(manipulation)
    entity.components.set(GroundingShadowComponent(castsShadow: true))
}

See Also

RealityKit and Reality Composer Pro

• Reality Composer Pro
• Chaparral Village: Building an immersive visionOS adventure game
• Designing no-code games with Reality Composer Pro 3
• Petite Asteroids: Building a volumetric visionOS game
• BOT-anist
• Swift Splash
• Diorama
• Building an immersive media viewing experience
• Enabling video reflections in an immersive environment
• Combining 2D and 3D views in an immersive app
• Understanding the modular architecture of RealityKit
• Using transforms to move, scale, and rotate entities
• Capturing screenshots and video from Apple Vision Pro for 2D viewing
• Implementing object tracking in your app
• Placing entities using head and device transform