Pathwren Crawler & robots.txt Tools FavIcon

Pathwren Crawler & robots.txt Tools

Six keyless GET endpoints about crawlers and robots.txt: verify that an IP really belongs to the crawler it claims to be, lint a robots.txt, decide one path for one crawler under RFC 9309, audit which AI crawlers a robots.txt really blocks, classify a User-Agent, and say whether a crawler can be verified at all. No account, no key, no handshake; open CORS, strong ETag, CC0. Rebuilt every six hours from the operators own published docs and IP ranges.

Visit API Docs

API Explorer

Interactive OpenAPI documentation. Requests are sent directly from your browser to the API provider.

View spec

Features & Specifications

HTTPS Support
Yes
CORS Support
Yes
Authentication
No
Pricing Model
Free

Similar APIs

View category
#1
Proof Random API

Proof Random API

116

Free drand quicknet relay for agent experiments. GET /v1/random with a required nonce and optional max returns ...

No Free
#2
Ciprand

Ciprand

7,777

Secure random string generator

No Unknown
#3
Wandbox

Wandbox

2,252

Code compiler supporting 35+ languages mentioned at wandbox.org

No Unknown
#4
Webcrawler API

Webcrawler API

1,626

Extract content from websites with powerful API. Do you want to get the website content? We can help you wi ...

apiKey Unknown
#5
Bulk Domain And Whois

Bulk Domain And Whois

1,465

Domain availability check. All tlds supported. Works in two modes: - single domain query: optimized for lat ...

rapidapi-key Free Option

User Comments

Please Sign in to leave a comment.

Be the first to share your thoughts!

Pathwren Crawler & robots.txt Tools
Pathwren Crawler & robots.txt Tools preview interface