AI crawler access check
Enter a URL. We read its robots.txt and show whether the AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) are allowed in, and which rules block them.
Recent checks
Only robots.txt is read. Nothing about your page is stored on our server.
Know which rule is standing in the way.看清是哪条规则挡住了访问。
Check access before rewriting your content. See how the declared crawler rules apply to the URL you care about.先确认访问规则,再投入内容优化。围绕你关心的网址,查看不同爬虫对应的允许与阻止规则。
Path-level context按路径判断
A rule for one folder can differ from the rest of the site.同一网站的某个目录,可能与其他页面采用不同规则。
Crawler-by-crawler view分爬虫查看
Review the relevant user-agent instead of guessing from one general rule.对照具体 user-agent,避免仅凭一条通用规则判断。
A focused handoff修改目标明确
Give your developer the URL, crawler and rule to review.把网址、爬虫和待核对规则一起交给开发人员。
ILLUSTRATIVE EXAMPLE示例场景
From a question to a useful next step从一个问题,走向有用的下一步
The starting point遇到的问题
A product page looks public, but its folder is disallowed in robots.txt.产品页能在浏览器打开,但所在目录在 robots.txt 中被禁止访问。
What to do如何处理
Review the matching path rule and the site's intended access policy.核对匹配的目录规则,并确认网站原本希望开放给哪些爬虫。
What to verify接着核对
Recheck the same URL after publishing the intended rule change.发布符合预期的规则后,使用同一网址复查。
A little more clarity使用前,再了解一点
Does an allowed result mean my page will be cited?允许访问就代表页面会被引用吗?
It describes the robots.txt rule. Successful fetching, indexing and citations are separate checks.它描述的是 robots.txt 规则;能否成功抓取、被收录或被引用,需要分别核对。
How should I interpret the results?应该怎样理解检查结果?
Use the evidence with the purpose of your page. Our scoring method explains what the checks measure and how to review their limits.结合页面用途核对具体依据。“评分方法”说明了检查衡量什么,以及如何理解其边界。 Read the method →查看评分方法 →
Put the next step into practice.把下一步,付诸实践。
The tool is ready at the top of this page.回到页面上方,即可使用当前工具。