All guides

Is Your robots.txt Blocking AI Crawlers? A Careful Guide

Understand robots.txt, provider-specific crawler documentation, and the limits of access controls before changing a production site's crawl policy.

By Ajredin “Dean” Bislimi · 2025-02-20

robots.txt gives compliant crawlers instructions about which paths they may request. Review it alongside noindex, authentication, and server rules; changing it does not guarantee indexing, citation, or traffic.

What robots.txt actually controls

A robots.txt file is a plain-text set of instructions at a site's root. It can tell compliant crawlers which paths they may request. Google's documentation makes clear that robots.txt is not an access-control system and does not guarantee that a URL will be removed from search results.

A page can also be affected by noindex directives, login requirements, server errors, canonicalization, or a site's internal links. Checking only robots.txt can therefore produce a false diagnosis. Review the complete publishing path.

Robots rules are path-specific and syntax-sensitive. Test a proposed change, keep a record of why it exists, and avoid broad disallows that accidentally block CSS, JavaScript, or important public pages needed for rendering.

OpenAI and Perplexity crawler choices

OpenAI publishes documentation for its crawlers and their purposes, including search-related discovery and user-requested access. Perplexity separately documents PerplexityBot. Use those official pages for current user-agent names and policy choices; do not copy an old list from an SEO article.

Allowing a crawler may make a public page eligible for that crawler to request, while disallowing it expresses a publishing preference. Neither choice guarantees that a product retrieves, cites, summarizes, or recommends the page.

A business should decide based on its content goals, privacy expectations, licensing position, and platform documentation. Public customer information should be accurate regardless of the choice. Private or sensitive information should be protected with real access control rather than robots.txt.

A safe review process

Inventory the public pages that should be discoverable. Check robots.txt, response status, redirects, noindex directives, canonical URLs, internal links, and whether the page works without authentication. Record the test date and the rule that produced the result.

Read the current OpenAI and Perplexity documentation before adding or changing user-agent groups. Make the smallest change that expresses the intended policy, then validate the file and monitor server behavior where appropriate.

If a business page is not appearing in an answer, do not assume robots.txt is the cause. First verify that the page exists, is useful, is linked, contains the relevant facts, and is eligible for the product's retrieval process. Report uncertainty explicitly.

Frequently asked questions

Does robots.txt control indexing?

No. robots.txt controls requests from compliant crawlers, while indexing and display involve other signals. Google recommends using appropriate mechanisms, such as noindex, when the goal is to keep a URL out of search results.

Does allowing GPTBot guarantee a ChatGPT citation?

No. Allowing a crawler addresses access only. It does not guarantee that a page is retrieved, selected, cited, or shown in a ChatGPT answer.

Is robots.txt a way to protect private information?

No. Use authentication and real access controls for private information. robots.txt is a public instruction for compliant crawlers, not a security boundary.

Sources and further reading