Read robots.txt by purpose and path按用途与路径理解 robots.txt
Separate search access from training preferences.区分搜索访问和训练偏好。
GEO Teacher · Updated 13 October 2026GEO Teacher · 更新于 2026 年 10 月 13 日
A path rule is not a whole-site verdict路径规则不等于全站结论
Disallow: / applies to the whole site for the applicable agent group. Disallow: /wp-admin/ applies to that path. Allow rules, wildcards and agent groups need to be considered together for the actual URL being checked.Disallow: / 对适用代理组表示全站禁止；Disallow: /wp-admin/ 表示该路径禁止。应针对实际检测网址综合判断 Allow、通配符及代理组。
Evaluate the matching group判断适用规则组
A specific matching agent group takes precedence over the wildcard fallback. Combine matching groups and use the most specific matching path rule; equally specific Allow and Disallow matches favor Allow. Empty Disallow values do not block everything.特定匹配代理组优先于通用组回退；合并匹配组后使用最具体的路径规则。Allow 与 Disallow 同样具体时优先 Allow。空的 Disallow 不表示禁止全站。
Choose the purpose按用途选择
Some robots identify search retrieval, others identify training or user-triggered requests. Their policies can differ. Verify a provider’s current documentation and decide according to your content rights and desired visibility, not a universal “allow all” template.不同标识可用于搜索检索、训练或用户触发请求，政策可能不同。应核对服务商当前文档，根据内容权利和可见度目标决定，不套用统一“允许全部”模板。
Know the limits了解边界
robots.txt is a voluntary crawling protocol, not access authorization or proof of an actual visit. Server restrictions can still prevent retrieval. An error reading robots.txt should be reported as uncertainty, not automatically as a perfect score. RFC 9309 defines the protocol.robots.txt 是爬取协议，不是访问授权，也不能证明实际访问。服务器限制仍可能阻止检索。读取文件出错应说明不确定性，而不是自动给满分。RFC 9309 定义了该协议。