Contents

SE-0525: Safe loading API for `RawSpan`

Summary of changes

We introduce a set of safe API to load and store values of certain safe types from the memory represented by RawSpan, MutableSpan, MutableRawSpan and OutputRawSpan instances. This will bolster the value of Swift in contexts where a process needs the ability to send data to other running processes via untyped buffers, as well as provide a set of building blocks for parsing utilities.

Motivation

In [SE-0447][SE-0447], we introduced RawSpan along with some unsafe functions to load values of arbitrary types. While it is safe to load some types with those functions, for example the native integer types, the unsafe annotation introduces an element of doubt for users of the standard library. This proposal aims to provide clarity for safe uses of byte-loading operations. In order to define safe byte-loading operations, we also need to clarify what are safe byte-storing operations, and this proposal tackles those as well.

Proposed solution

We propose two new protocols, supporting the conversion between initialized typed values and initialized raw bytes. The first will be conformed to by types that can always safely be read as raw bytes: ConvertibleToBytes. The second will be conformed to by types that can always be safely interpreted from raw bytes: ConvertibleFromBytes.

ConvertibleToBytes

When initializing memory to any value of a type conforming to ConvertibleToBytes, every byte underlying the type's stride must be initialized.

@_marker protocol ConvertibleToBytes: Copyable {}

A type can conform to ConvertibleToBytes if its memory representation includes no padding. In other words, the sum of the size of its stored properties is equal to its stride. For example, an Optional<Int16> is stored in 3 bytes out of a stride of 4, and therefore Optional<Int16> cannot conform. A struct Pair { var a, b: Int16 } could conform to ConvertibleToBytes, as its size and stride are equal.

A type that conforms to ConvertibleToBytes must have:

  • one or more stored properties,
  • all of its stored properties have types conforming to ConvertibleToBytes,
  • its stored properties are stored contiguously in memory, with no padding,
  • none of its values disregards a subset of its bytes (this makes most enums ineligible.)

Many basic types in the standard library will conform to this protocol, but types outside the standard library will not initially be able to conform to ConvertibleToBytes.

A conformance to ConvertibleToBytes can only be declared by a type's containing module.

ConvertibleFromBytes
@_marker protocol ConvertibleFromBytes: BitwiseCopyable {}

A type can conform to ConvertibleFromBytes if every bit pattern for every byte of its stored properties is valid. Note that this allows conformances for types with internal or trailing padding. A conformer to ConvertibleFromBytes must not have semantic constraints on the values of its stored properties. All its stored properties must themselves conform to ConvertibleFromBytes.

For example, a type representing two-dimensional Cartesian coordinates, such as struct Point { var x, y: Int } could conform to ConvertibleFromBytes. Its stored properties are Int, which is ConvertibleFromBytes. There are no semantic constraints between the x and y properties: any combination of Int values can represent a valid Point.

In contrast, Range<Int> could not conform to ConvertibleFromBytes, even though on the surface it has the same composition as Point. There is a semantic constraint between the two stored properties of Range: lowerBound must be less than or equal to upperBound. This makes it unable to conform to ConvertibleFromBytes.

Other examples of types that cannot conform to ConvertibleFromBytes are UnicodeScalar (some bit patterns are invalid,) a hypothetical UTF8-encoded SmallString (the sequencing of the constituent bytes matters for validity,) and UnsafeRawPointer. The case of pointers is illuminating: the semantic validity of a value is unknown until runtime, since the runtime environment determines the actual set of valid values.

The compiler cannot enforce the semantic requirements of ConvertibleFromBytes, therefore types outside the standard library can only conform with an unchecked conformance.

extension MyType: @unchecked ConvertibleFromBytes {}

A conformance to ConvertibleFromBytes can only be declared by a type's containing module.

FullyInhabited
typealias FullyInhabited = ConvertibleToBytes & ConvertibleFromBytes

FullyInhabited is the intersection of ConvertibleToBytes and ConvertibleFromBytes.

RawSpan and MutableRawSpan

RawSpan and MutableRawSpan will have a new, generic load(as:) function that returns ConvertibleFromBytes values read from the underlying memory, with no pointer-alignment restriction. Because the returned values are ConvertibleFromBytes and the request is bounds-checked, this load(as:) function is safe.

extension RawSpan {
  func load<T: ConvertibleFromBytes>(
    fromByteOffset: Int,
    as: T.Type = T.self
  ) -> T
}

Additionally, a special version of load(as:) will have an additional argument to control the byte order of the value being loaded, for values of types conforming to both ConvertibleFromBytes and FixedWidthInteger:

extension RawSpan {
  func load<T: ConvertibleFromBytes & FixedWidthInteger>(
    fromByteOffset: Int,
    as: T.Type = T.self,
    _ byteOrder: ByteOrder
  ) -> T
}

@frozen
enum ByteOrder: Equatable, Hashable, Sendable {
  case bigEndian, littleEndian
  
  static var native: Self { get }
}

The list of standard library types to conform to ConvertibleFromBytes & FixedWidthInteger is UInt8, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, UInt, Int, UInt128, and Int128.

The load(as:) functions are not atomic operations.

The load(as:) functions will not have equivalents with unchecked byte offset. If that functionality is needed, the unsafeLoad(fromUncheckedByteOffset:as:) function is already available.

Subscripts for the RawSpan family

As a convenience for the specific case of UInt8, we will define subscripts for RawSpan, MutableRawSpan and OutputRawSpan, similar to the existing Span, MutableSpan and OutputSpan subscripts:

extension RawSpan {
  subscript(_ byteOffset: Int) -> UInt8 { get }
  
  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get }
}

extension MutableRawSpan {
  subscript(_ byteOffset: Int) -> UInt8 { get set }

  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get set }
}

extension OutputRawSpan {
  subscript(_ byteOffset: Int) -> UInt8 { get set }

  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get set }
}
MutableRawSpan and OutputRawSpan

MutableRawSpan will gain new overloads of storeBytes():

extension MutableRawSpan {
  mutating func storeBytes<T>(
    of value: T,
    toByteOffset offset: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
  
  @unsafe
  mutating func storeBytes<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type
  ) where T: BitwiseCopyable

  mutating func storeBytes<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
}

The existing storeBytes function constrained to T: BitwiseCopyable will be marked @unsafe, as it is possible for compiler optimizations to result in uninitialized bytes. The addition of a repeating variant corrects an omission; it is also marked @unsafe.

OutputRawSpan will have matching append() functions:

extension OutputRawSpan {
  mutating func append<T>(
    _ value: T,
    as type: T.Type
  ) where T: ConvertibleToBytes & BitwiseCopyable

  mutating func append<T>(
    _ value: T,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
  
  mutating func append<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type
  ) where T: ConvertibleToBytes & BitwiseCopyable
  
  mutating func append<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
}

The existing append function constrained to just T: BitwiseCopyable will be marked @unsafe.

Loading and storing in-memory types

The memory layout of many of the types eligible for ConvertibleFromBytes and/or ConvertibleToBytes is not guaranteed to be stable across compiler and library versions. This is not an issue for the use case envisioned for these API, where data is sent among running processes, or stored for later use by the same process. For more elaborate needs such as serializing for network communications or file system storage, the API proposed here can only be considered as a building block.

Span and MutableSpan

Span will have a new initializer init(viewing: RawSpan) to allow viewing a range of untyped memory as a typed Span, when Span.Element conforms to ConvertibleFromBytes. These conversions will check for alignment and bounds. When the RawSpan's pointer alignment is incorrect for Element, this initializer will trap. When the bounds are not a multiple of the stride, this initializer will trap.

extension Span {
  @_lifetime(copy bytes)
  init(viewing bytes: RawSpan) where Element: ConvertibleFromBytes
}

MutableSpan will have new initializers to mutate the memory of a MutableRawSpan as a typed MutableSpan, when its Element conforms to ConvertibleToBytes & ConvertibleFromBytes. These conversions will check for alignment and bounds. When the MutableRawSpan's pointer alignment is incorrect for Element, these initializers will trap. When the bounds are not a multiple of the stride, these initializers will trap.

extension MutableSpan {
  @_lifetime(&mutableBytes)
  init(mutating mutableBytes: inout MutableRawSpan)
    where Element: ConvertibleToBytes & ConvertibleFromBytes

  @_lifetime(copy mutableBytes)
  init(mutableBytes: consuming MutableRawSpan)
    where Element: ConvertibleToBytes & ConvertibleFromBytes
}

The conversions from RawSpan to Span only support well-aligned views with the native byte order. The [swift-binary-parsing][swift-binary-parsing] package provides a more fully-featured ParserSpan type for use cases beyond reinterpreting memory in-place. We expect a future proposal to include functionality to help determine the memory alignment of a RawSpan instance.

The existing bytes and mutableBytes accessors will have safe overloads for when Element conforms to ConvertibleToBytes (bytes accessor) and to ConvertibleToBytes & ConvertibleFromBytes (mutableBytes accessor).

Detailed design

ConvertibleToBytes

A ConvertibleToBytes type has at least one stored property, and all its stored properties are values of ConvertibleToBytes types.

The memory representation of a ConvertibleToBytes type must include no padding. For example, struct A { var v: [3 of Int8]; var n: Int64 } has two stored properties, both ConvertibleToBytes, but it has five bytes of padding. <!-- MemoryLayout<A>.stride - (MemoryLayout<[3 of Int8]>.stride + MemoryLayout<Int64>.stride) equals 5 -->

@_marker protocol ConvertibleToBytes: Copyable {}

Custom types will not be allowed to declare a conformance to ConvertibleToBytes at this time.

ConvertibleFromBytes

A ConvertibleFromBytes type has a valid value for every bit pattern of every byte of its stored properties. This precludes any semantic constraints, whether static or dynamic, on the values of a type conforming to ConvertibleFromBytes.

@_marker protocol ConvertibleFromBytes: BitwiseCopyable {}

Custom types will be allowed to declare an unchecked conformance to ConvertibleFromBytes.

Types that do not fully use a byte, such as Bool, are disallowed. Undefined behaviour can result when an invalid bit pattern is interpreted as such a value.

ByteOrder
@frozen
enum ByteOrder: Equatable, Hashable, Sendable {
  /// Bytes are ordered with the most significant bits
  /// starting at the lowest memory address.
  case bigEndian
  
  /// Bytes are ordered with the least significant bits
  /// starting at the lowest memory address.
  case littleEndian

  /// The native byte ordering for the runtime target.
  static var native: Self { get }
}
RawSpan
extension RawSpan {
  /// Returns a value constructed from the raw memory at the specified offset.
  ///
  /// The range of bytes required to construct a value of type `T` starting at
  /// `offset` must be completely within the span.
  /// `offset` is not required to be aligned for `T`.
  ///
  /// - Parameters:
  ///   - offset: The offset from the beginning of this span, in bytes.
  ///     `offset` must be nonnegative.
  ///   - type: The type of the instance to create.
  /// - Returns: A new value of type `T`, read from `offset`.
  func load<T: ConvertibleFromBytes>(
    fromByteOffset offset: Int,
    as type: T.Type = T.self
  ) -> T

  /// Returns a value constructed from the raw memory at the specified offset.
  ///
  /// The range of bytes required to construct a value of type `T` starting at
  /// `offset` must be completely within the span.
  /// `offset` is not required to be aligned for `T`.
  ///
  /// - Parameters:
  ///   - offset: The offset from the beginning of this span, in bytes.
  ///     `offset` must be nonnegative.
  ///   - type: The type of the instance to create.
  ///   - byteOrder: The order in which the bytes will be decoded.
  /// - Returns: A new value of type `T`, read from `offset`.
  func load<T: ConvertibleFromBytes & FixedWidthInteger>(
    fromByteOffset offset: Int,
    as type: T.Type = T.self,
    _ byteOrder: ByteOrder
  ) -> T
  
  /// Accesses the byte at the specified offset in the span.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  subscript(_ byteOffset: Int) -> UInt8 { get }

  /// Accesses the byte at the specified offset in the span.
  ///
  /// This subscript does not validate `byteOffset`. Using this subscript
  /// with an invalid `byteOffset` results in undefined behaviour.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get }
  
  /// View a typed span as a raw span.
  @_lifetime(copy elements)
  init<T: ConvertibleToBytes>(elements: consuming Span<T>)
  
  /// Unsafely view a typed span as a raw span.
  @unsafe
  @_lifetime(copy unsafeElements)
  init<T>(unsafeElements: consuming Span<T>)  
}
MutableRawSpan
extension MutableRawSpan {
  /// Stores the given value's bytes to the specified offset into
  /// the span's memory.
  ///
  /// The range of bytes required to store a value of type `T` starting at
  /// byte offset `offset` must be completely within the span.
  /// `offset` is not required to be aligned for `T`.
  ///
  /// - Parameters:
  ///   - value: The value to store as raw bytes.
  ///   - offset: The offset in bytes into the span's memory at which to begin
  ///       writing the bytes from the value.
  ///   - type: The type of the instance to store.
  ///   - byteOrder: The order in which the bytes will be encoded to the span.
  mutating func storeBytes<T>(
    of value: T,
    toByteOffset offset: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger

  /// Stores the given value's bytes repeatedly into this span's memory.
  ///
  /// There must be at least `count * MemoryLayout<T>.stride` bytes
  /// available in the span.
  ///
  /// - Parameters:
  ///   - repeatedValue: The value to store as raw bytes.
  ///   - count: The number of copies of `repeatedValue` to store
  ///      into this span.
  ///   - type: The type of the instance to store repeatedly.
  @unsafe
  mutating func storeBytes<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type
  ) where T: BitwiseCopyable

  /// Stores the given value's bytes repeatedly into this span's memory.
  ///
  /// There must be at least `count * MemoryLayout<T>.stride` bytes
  /// available in the span.
  ///
  /// - Parameters:
  ///   - repeatedValue: The value to store as raw bytes.
  ///   - count: The number of copies of `repeatedValue` to store
  ///      into this span.
  ///   - type: The type of the instance to store repeatedly.
  ///   - byteOrder: The order in which the bytes will be encoded to the span.
  mutating func storeBytes<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger

  /// Returns a value constructed from the raw memory at the specified offset.
  ///
  /// The range of bytes required to construct a value of type `T` starting at
  /// `offset` must be completely within the span.
  /// `offset` is not required to be aligned for `T`.
  ///
  /// - Parameters:
  ///   - offset: The offset from the beginning of this span, in bytes.
  ///     `offset` must be nonnegative.
  ///   - type: The type of the instance to create.
  /// - Returns: A new value of type `T`, read from `offset`.
  func load<T: ConvertibleFromBytes>(
    fromByteOffset offset: Int,
    as type: T.Type = T.self
  ) -> T

  /// Returns a value constructed from the raw memory at the specified offset.
  ///
  /// The range of bytes required to construct a value of type `T` starting at
  /// `offset` must be completely within the span.
  /// `offset` is not required to be aligned for `T`.
  ///
  /// - Parameters:
  ///   - offset: The offset from the beginning of this span, in bytes.
  ///     `offset` must be nonnegative.
  ///   - type: The type of the instance to create.
  ///   - byteOrder: The order in which the bytes will be decoded.
  /// - Returns: A new value of type `T`, read from `offset`.
  func load<T: ConvertibleFromBytes & FixedWidthInteger>(
    fromByteOffset offset: Int,
    as type: T.Type = T.self,
    _ byteOrder: ByteOrder
  ) -> T

  /// Accesses the byte at the specified offset in the span.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  subscript(_ byteOffset: Int) -> UInt8 { get set }

  /// Accesses the byte at the specified offset in the span.
  ///
  /// This subscript does not validate `byteOffset`. Using this subscript
  /// with an invalid `byteOffset` results in undefined behaviour.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get set }
  
  /// Mutate the elements of a typed span as bytes.
  @_lifetime(&mutableSpan)
  init<T>(mutating mutableSpan: inout MutableSpan<T>)
    where T: ConvertibleToBytes & ConvertibleFromBytes

  /// Convert a typed span to a raw span.
  @_lifetime(copy elements)
  init<T>(elements: consuming MutableSpan<T>)
    where T: ConvertibleToBytes & ConvertibleFromBytes
  
  /// Unsafely convert a typed span to a raw span.
  @unsafe
  @_lifetime(copy unsafeElements)
  init<T>(unsafeElements: consuming MutableSpan<T>)
}
OutputRawSpan
extension OutputRawSpan {
  /// Appends the given value's bytes to this span's bytes.
  ///
  /// There must be at least `MemoryLayout<T>.size` bytes available
  /// in the span.
  ///
  /// - Parameters:
  ///   - value: The value to store as raw bytes.
  ///   - type: The type of the instance to store.
  mutating func append<T>(
    _ value: T,
    as type: T.Type
  ) where T: ConvertibleToBytes & BitwiseCopyable

  /// Appends the given value's bytes to this span's bytes.
  ///
  /// There must be at least `MemoryLayout<T>.size` bytes available
  /// in the span.
  ///
  /// - Parameters:
  ///   - value: The value to store as raw bytes.
  ///   - type: The type of the instance to store.
  ///   - byteOrder: The order in which the bytes will be encoded to the span.
  mutating func append<T>(
    _ value: T,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
  
  /// Appends the given value's bytes repeatedly to this span's bytes.
  ///
  /// There must be at least `count * MemoryLayout<T>.stride` bytes
  /// available in the span.
  ///
  /// - Parameters:
  ///   - repeatedValue: The value to store as raw bytes.
  ///   - count: The number of copies of `repeatedValue` to append
  ///       to this span.
  ///   - type: The type of the instance to store repeatedly.
  mutating func append<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type
  ) where T: ConvertibleToBytes & BitwiseCopyable

  /// Appends the given value's bytes repeatedly to this span's bytes.
  ///
  /// There must be at least `count * MemoryLayout<T>.stride` bytes
  /// available in the span.
  ///
  /// - Parameters:
  ///   - repeatedValue: The value to store as raw bytes.
  ///   - count: The number of copies of `repeatedValue` to append
  ///       to this span.
  ///   - type: The type of the instance to store repeatedly.
  ///   - byteOrder: The order in which the bytes will be encoded to the span.
  mutating func append<T>(
    repeating repeatedValue: T,
    count: Int,
    as type: T.Type,
    _ byteOrder: ByteOrder
  ) where T: ConvertibleToBytes & BitwiseCopyable & FixedWidthInteger
  
  /// Accesses the byte at the specified offset in the span.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  subscript(_ byteOffset: Int) -> UInt8 { get set }

  /// Accesses the byte at the specified offset in the span.
  ///
  /// This subscript does not validate `byteOffset`. Using this subscript
  /// with an invalid `byteOffset` results in undefined behaviour.
  ///
  /// - Parameter byteOffset: The offset of the byte to access. `byteOffset`
  ///     must be greater than or equal to zero, and less than `byteCount`.
  @unsafe
  subscript(unchecked byteOffset: Int) -> UInt8 { get set }

  /// The offsets valid for subscripting the span, in ascending order.
  public var byteOffsets: Range<Int>
}
Span
extension Span {
  /// View initialized raw memory as a typed span.
  ///
  /// The `byteCount` of `bytes` must be a multiple of `Element`'s stride,
  /// and the starting address of `bytes` must be well-aligned for the type
  /// of `Element`. If either of these requirements is not met, this initializer
  /// will trap at runtime.
  @_lifetime(copy bytes)
  init(viewing bytes: RawSpan) where Element: ConvertibleFromBytes
}

extension Span where Element: ConvertibleToBytes {
  /// A raw span over the memory represented by this span.
  ///
  /// - Returns: A RawSpan over the memory represented by this span.
  @_lifetime(copy self)
  var bytes: RawSpan { get }
}
MutableSpan
extension MutableSpan {
  /// Mutate untyped memory as a typed span.
  ///
  /// The `byteCount` of `mutableBytes` must be a multiple of `Element`'s stride,
  /// and the starting address of `mutableBytes` must be well-aligned for
  /// the type of `Element`. If either of these requirements is not met,
  /// this initializer will trap at runtime.
  @_lifetime(&mutableBytes)
  init(mutating mutableBytes: inout MutableRawSpan)
    where Element: ConvertibleToBytes & ConvertibleFromBytes

  /// Convert a raw span to a typed span.
  ///
  /// The `byteCount` of `mutableBytes` must be a multiple of `Element`'s stride,
  /// and the starting address of `mutableBytes` must be well-aligned for
  /// the type of `Element`. If either of these requirements is not met,
  /// this initializer will trap at runtime.
  @_lifetime(copy mutableBytes)
  init(mutableBytes: consuming MutableRawSpan)
    where Element: ConvertibleToBytes & ConvertibleFromBytes
}

extension MutableSpan where Element: ConvertibleToBytes & ConvertibleFromBytes {
  /// A mutable raw span over the memory represented by this span.
  ///
  /// - Returns: A MutableRawSpan over the memory represented by this span.
  @_lifetime(&self)
  var mutableBytes: MutableRawSpan { mutating get }
}
Conformances in the standard library

The following conformances will be implemented in the standard library, depending on the platform availability of the base types:

extension UInt8:   ConvertibleToBytes, ConvertibleFromBytes {}
extension Int8:    ConvertibleToBytes, ConvertibleFromBytes {}
extension UInt16:  ConvertibleToBytes, ConvertibleFromBytes {}
extension Int16:   ConvertibleToBytes, ConvertibleFromBytes {}
extension UInt32:  ConvertibleToBytes, ConvertibleFromBytes {}
extension Int32:   ConvertibleToBytes, ConvertibleFromBytes {}
extension UInt64:  ConvertibleToBytes, ConvertibleFromBytes {}
extension Int64:   ConvertibleToBytes, ConvertibleFromBytes {}
extension UInt:    ConvertibleToBytes, ConvertibleFromBytes {}
extension Int:     ConvertibleToBytes, ConvertibleFromBytes {}

extension UInt128: ConvertibleToBytes, ConvertibleFromBytes {}
extension Int128:  ConvertibleToBytes, ConvertibleFromBytes {}

extension Float16: ConvertibleToBytes, ConvertibleFromBytes {}
extension Float32: ConvertibleToBytes, ConvertibleFromBytes {} // `Float`
extension Float64: ConvertibleToBytes, ConvertibleFromBytes {} // `Double`

extension Duration: ConvertibleToBytes, ConvertibleFromBytes {}

extension InlineArray: ConvertibleToBytes
  where Element: ConvertibleToBytes {}
extension InlineArray: ConvertibleFromBytes
  where Element: ConvertibleFromBytes {}

extension CollectionOfOne: ConvertibleToBytes
  where Element: ConvertibleToBytes {}
extension CollectionOfOne: ConvertibleFromBytes
  where Element: ConvertibleFromBytes {}

extension ClosedRange: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}
extension Range: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}

extension PartialRangeFrom: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}
extension PartialRangeFrom.Iterator: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}
extension PartialRangeThrough: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}
extension PartialRangeUpTo: ConvertibleToBytes
  where Bound: ConvertibleToBytes {}

extension Bool: ConvertibleToBytes {}
extension ObjectIdentifier: ConvertibleToBytes {}

extension UnsafePointer: ConvertibleToBytes {}
extension UnsafeMutablePointer: ConvertibleToBytes {}
extension UnsafeRawPointer: ConvertibleToBytes {}
extension UnsafeMutableRawPointer: ConvertibleToBytes {}
extension OpaquePointer: ConvertibleToBytes {}

extension UnsafeBufferPointer: ConvertibleToBytes {}
extension UnsafeMutableBufferPointer: ConvertibleToBytes {}
extension UnsafeRawBufferPointer: ConvertibleToBytes {}
extension UnsafeMutableRawBufferPointer: ConvertibleToBytes {}

Note: any of the types in the list above which is missing a prerequisite BitwiseCopyable conformance will gain one.

Top-level safe bitCast function

With the two protocols we have defined, we gain the ability to define a safe function to reinterpret types:

/// Returns the bits of the given instance, interpreted as having the specified
/// type.
///
/// `T` and `U` must have the same-sized memory representation.
/// If they don't, this function will trap.
///
/// - Parameters:
///   - original: The instance to cast to `type`.
///   - type: The type to cast `original` to.
/// - Returns: A new instance of type `U`, cast from `original`.
func bitCast<T, U>(_ original: T, to type: U.Type) -> U
  where T: ConvertibleToBytes, U: ConvertibleFromBytes

Source compatibility

This proposal consists only of additions, and should therefore be source compatible. Adding overloads is risky, and it is possible these additions might affect some existing valid code. Testing is required to rule out significant compatibility issues.

ABI compatibility

The functions in this proposal will be implemented in such a way as to avoid creating additional ABI.

These functions require the existence of Span, and have a minimum deployment target on Darwin-based platforms, where the Swift standard library is distributed with the operating system.

ByteOrder is a new type and, as such, will come with availability. The functions that use a ByteOrder argument will share that same availability.

Implications on adoption

The additions described in this proposal require a new version of the Swift standard library.

Future directions

Validation for the ConvertibleToBytes protocol

ConvertibleToBytes conformances will undergo additional validation by the compiler at a later time. This protocol can be fully validated at compilation time, since it relies entirely on the layout of the type in addressable memory. It should be possible to automate ConvertibleToBytes conformances in a manner similar to BitwiseCopyable.

Alongside validation, we could consider automatically inserting stored null bytes instead of padding for types which elect it.

Partial validation for the ConvertibleFromBytes protocol

ConvertibleFromBytes conformances may undergo some validation by the compiler at a later time. The compiler can enforce that all of a type's stored properties conform to ConvertibleFromBytes. It cannot directly enforce the absence of semantic constraints on the type's stored properties, but we may choose to accept a roundabout way of supporting its absence, such as if all the stored properties are public and mutable (var bindings).

Support for types imported from C

The Clang importer could be taught which basic C types support these protocols. It would be useful to have a way to declare a conformance to these protocols for C types which are aggregates. For example, we could relax the restriction that these conformances can only be declared in a type's containing module, for imported C types only.

Support for tuples and SIMD types

Tuples composed of ConvertibleToBytes types should themselves be ConvertibleToBytes. The same applies to ConvertibleFromBytes. The standard library's SIMD types also seem to be naturally suited to these protocols.

Utilities to examine the alignment of a RawSpan

The Span initializers require a correctly-aligned RawSpan; there should be utilities to identify the offsets that are well-aligned for a given type.

Renaming of @unsafe functions and properties that do not explicitly include an unsafety marker

Some functions and properties introduced in earlier proposals have since been annotated as unsafe, but their names did not indicate unsafety. We should identify names that involve unsafety even when strict memory safety mode is disabled. We should plan carefully for the renaming of these symbols, and doing so in a separate proposal will be most conducive to a successful and minimally disruptive outcome.

Appending to OutputSpan using OutputRawSpan and vice-versa

The proposal originally included a function to append to a portion of an OutputSpan using an OutputRawSpan when Element conforms to ConvertibleFromBytes:

span.append(upTo: capacity) { for _ in 0..<$0.capacity { $0.append(UInt8.zero) } }

The argument label for the capacity (here "upTo") is a difficult topic. We note that this shape of API is also necessary on UniqueArray, and applies to insert and replace as well as append. We should discuss the naming of these API in a context where they will get sufficient attention, namely SE-0527 (UniqueArray and RigidArray), and once that is settled we can propose to add them to OutputSpan and OutputRawSpan, along with appropriate variants.

Alternatives considered

Encoding the name of the type being loaded into the function names

Having a series of concrete functions such as loadInt32(fromByteOffset::) and storeBytes(int32:toByteOffset::) would be easier on the type checker, by avoiding the problem of overloaded symbols.

Waiting for a compiler-validated ConvertibleToBytes layout constraint

The need for the functionality in this proposal is urgent and can be achieved with standard library additions. Validation of the ConvertibleToBytes layout constraint will require significant compiler work, and we believe that the API as proposed in this document will provide significant value on its own.

Making these additions generic over FixedWidthInteger & BitwiseCopyable and BinaryFloatingPoint & BitwiseCopyable

These are not sufficient constraints, since none of these three protocols mandate that their conformers must be fully inhabited.

Adding only a FullyInhabited protocol instead of the separate ConvertibleToBytes and ConvertibleFromBytes protocols.

The second pitch for this proposal proposed only FullyInhabited. The discussions showed that the pair of protocols would eventually be needed, and that the implementation burden would be similar. Implementing the pair of protocols seems to be the better option.

Omitting the ByteOrder parameters

The standard library's FixedWidthInteger protocol includes computed properties var .bigEndian: Self and var .littleEndian: Self. These could be used to modify the arguments of storeBytes(_:as:), or to modify the return values from load(as:). Unfortunately these properties are not clear, because they return Self, conflating byte ordering with otherwise valid values. This proposal applies the consideration of byte ordering to the operation where it belongs: serialization.

Defaulting to aligned operations for the safe load() functions

UnsafeRawPointer's original load() function requires correct alignment, and the less restrictive loadUnaligned() was added later. We have long considered this unfortunate, and this proposal seeks to improve on the status quo by making our new safe load() functions perform unaligned operations.

Defaulting toByteOffset and fromByteOffset parameters to a value of zero

The proposal originally included defaulted parameter values for these parameters for the new load and storeBytes functions. It was pointed out that defaulted parameters may give the impression that loading or storing a value using these functions must use the full length of the RawSpan or MutableRawSpan. Noting that defaulted parameters already exist on similar, pre-existing unsafe API, we may need to harmonize the decision at a later time.

Acknowledgments

Thanks to Karoy Lorentey, Nate Cook, and Stephen Canon for taking the time to discuss this topic.

Enums to represent the byte order have previously been pitched by Michael Ilseman (Unicode Processing APIs) and by YOCKOW (ByteOrder type).