Contents

ptsochantaris/akashic-table

Akashic Table

Overview

I needed a way to keep a very large list of small, uniform records on disk and search it quickly by identifier, without paying for serialisation or holding the whole thing in memory. AkashicTable maps a file directly into memory and stores its elements as raw bytes inside it, so reads and writes go straight to the mapping with no copy or decode step, and the contents persist between launches.

  • Elements conform to RowIdentifiable (they expose an Int64 rowId) and must be BitwiseCopyable, since they are written to and read from the mapping as raw bytes.
  • The collection is always kept sorted in ascending rowId order, so lookups, insertions, and replacements use a binary search internally.
  • It conforms to RandomAccessCollection and MutableCollection, so it works with the standard collection APIs, as well as ContiguousBytes for direct access to its backing storage.
  • The backing file grows automatically, page-aligned, as more elements are added.

Usage

import AkashicTable

struct LogEntry: RowIdentifiable, BitwiseCopyable {
    let rowId: Int64
    var code: Int32
}

// Open (creating if needed) and map a backing file, sized for at least 100k records
let table = try AkashicTable<LogEntry>(at: "/path/to/log.dat", minimumCapacity: 100_000)

// Insert or replace by rowId, keeping the table sorted
try table.append(LogEntry(rowId: 1, code: 200))
try table.append(contentsOf: [
    LogEntry(rowId: 2, code: 404),
    LogEntry(rowId: 3, code: 500)
])

// Use it like any other collection
print(table.count) // 3
for entry in table {
    print(entry.rowId, entry.code)
}

// Remove elements by identifier, by predicate, or by position
table.deleteEntries(with: [2])
table.deleteAll { $0.code >= 500 }

// Flush to disk and release the mapping when done
table.shutdown()

// ...and map it again later, preserving all stored elements
try table.resume()

Pass validateOrder: true at initialisation to verify that an existing file's rows are in strictly ascending rowId order, and useCache: false to map the file with MAP_NOCACHE when its pages don't need to be retained in the buffer cache.

License

Released under the terms of the MIT license, see the LICENSE file for license rights and limitations (MIT).

Package Metadata

Repository: ptsochantaris/akashic-table

Default branch: main

README: README.md