Contents

ptsochantaris/can-proceed

A small, tested, no-frills parser of robots.txt files in Swift.

Quick example

let exampleRobotsFileLocation = URL(string: "https://www.bbc.co.uk/robots.txt")!
let robotsData = try! Data(contentsOf: exampleRobotsFileLocation)
let robotsText = String(data: robotsData, encoding: .utf8)!

let check = CanProceed.parse(robotsText)

let shouldProceed = check.agent("ChatGPT-User", canProceedTo: "/news")

guard shouldProceed else {
    complain()
    return
}

...

check.sitemaps                          // contains any sitemap URLs
check.agents["chatgpt-user"]?.crawlDelay // any crawl delay setting, per agent
check.host                              // any host directive

License

Copyright (c) 2024 Paul Tsochantaris. Licensed under the MIT License, see LICENSE for details.

Package Metadata

Repository: ptsochantaris/can-proceed

Default branch: main

README: README.md