This package provides a robots.txt parser for the Robots Exclusion Protocol in the Go programming language.

The implementation follows Google's Robots.txt Specification.

The code is simple and straightforward. The structs exposed by this package consist of basic data types only, making it easy to encode and decode them using one of Go's encoding packages. And though performance wasn't a design goal, this package should never become your program's bottleneck.

Run

go get github.com/slyrz/robots

robots := robots.New(file, "your-user-agent")
if robots.Allow("/some/path") {
	// Crawl it!
	// ...
}

robots is released under MIT license. You can find a copy of the MIT License in the LICENSE file.