--- title: "Set a deliberate robots.txt policy for AI crawlers" url: https://resahost.net/en/guides/robots-txt-for-ai-crawlers/ canonical: https://resahost.net/en/guides/robots-txt-for-ai-crawlers/ language: en description: "Set a deliberate AI crawler policy, distinguish search from training and verify access without mistaking robots.txt for website security." author: ResaHost author_url: https://resahost.net/en/contact/ related: - "AI search visibility Organisation Schema: Describe Your Business Accurately Read guide → — https://resahost.net/en/guides/organization-schema-for-your-business/" - "AI search visibility llms.txt Explained: An Optional Guide for Your Website Read guide → — https://resahost.net/en/guides/llms-txt-explained/" - "AI search visibility What Is AI Search Visibility? A Guide for Businesses Read guide → — https://resahost.net/en/guides/what-is-ai-search-visibility/" --- AI search visibility 5 min read · Updated 08/2026 # Set a deliberate robots.txt policy for AI crawlers Before changing crawler rules, decide what you want to permit: discovery of public service pages, use of content for model training, or visits initiated by a user. These are different purposes, and products can provide different controls for them. A blanket instruction to allow every AI bot is not a policy. Nor does blocking a training crawler necessarily remove a website from an AI search product. Document the intended outcome and use the current instructions for each provider you actually need to support. ## What robots.txt does Robots.txt is a public text file associated with a website's host. It communicates crawling rules to cooperating crawlers. It does not protect passwords, prevent malicious visitors or guarantee that a URL cannot appear in search. Google explains that crawling restrictions and indexing controls serve different purposes. [Google robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro). Private systems need authentication and access control. Do not list sensitive locations in robots.txt and assume the list makes them private. Equally, a crawler may need to fetch a page to see its noindex instruction, so combining controls without understanding them can defeat the intended result. ## Distinguish the main roles | Provider control | Purpose to evaluate | | --- | --- | | OAI-SearchBot | OpenAI search discovery | | GPTBot | OpenAI model-training crawl policy | | ChatGPT-User | Certain user-initiated page visits | | PerplexityBot | Perplexity search crawling | | Perplexity-User | User-initiated fetching | Consult the [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots/) and [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) before implementation. User-initiated fetchers can behave differently from automatic crawlers, so robots.txt should not be treated as a universal block on all product use. For other providers, check their current documentation rather than copying historical bot names. Search products, training controls and user agents change. A recognisable name in a log is also not proof of identity: user-agent strings can be impersonated. ## An example of a policy decision A company might decide to allow public search access while declining a particular training crawl. Express those decisions independently. Keep existing general crawler rules and required CMS exceptions intact; a small example copied from an article is not a safe replacement for your whole file. List the reasons for any exclusions. An internal preview should normally be protected at the application or server, while a public documentation area might intentionally be available. Ask content owners about licensing and contractual restrictions before adopting a broad policy. Our [ChatGPT discovery guide](https://resahost.net/en/guides/help-chatgpt-search-find-your-website/) explains the search use case, while [llms.txt](https://resahost.net/en/guides/llms-txt-explained/) describes an optional content index. Neither file grants a search ranking or changes the truth of published facts. ## Verify the complete access path Open the live robots.txt URL and make sure it returns the intended plain text, not a login page or a cached obsolete version. Review broad rules such as a full-site disallow and confirm whether they are intentional for that environment. Next test public destination pages. A CDN challenge, server error, geographic block or authentication requirement can prevent access even when robots.txt allows it. Ask the hosting team to inspect relevant responses and logs. Do not expose secret headers, credentials or private request bodies in a shared report. Where provider verification uses published network ranges, obtain the current ranges from the provider and maintain the rule. Avoid granting unrestricted privileged access simply because a request claims a bot name. Keep any exception limited to the public content and purpose it needs. ## Change carefully and record the result Save the previous configuration and note the reason for the change. Test in the site's normal release process, publish it, then verify the live response and important public pages. Cache expiry and provider refreshes mean external behaviour may not change instantly. Record the date and what you checked. Follow later with the [AI visibility checklist](https://resahost.net/en/guides/ai-search-visibility-checklist/) if discovery is the objective. A successful fetch confirms access at that time; it does not confirm indexing or inclusion in every answer. [ResaHost website security](https://resahost.net/en/services/website-security/) and the technical website owner should coordinate access rules so search requirements do not conflict with protective controls. Your website, taken care of. Hosting, domain, email and security in one agreement — from €69/month, excluding VAT. Choose hosting ## Frequently asked questions ## Can robots.txt stop hostile bots? No. Use appropriate server controls for security. ## Should I permit training to obtain search visibility? Review the provider's separate controls. Do not assume those permissions are inseparable. ## Who should own the file? Assign a technical owner, with content and legal input where rights or policy are involved. Review it when the website or provider policy changes. ![Olli Junes: Founder and CEO of ResaHost](https://resahost.net/assets/team/ollijunes.webp) Author Olli Junes: Founder and CEO of ResaHost Founder and CEO · ResaHost, Finland @resacolli LinkedIn ## Read next AI search visibility ### Organisation Schema: Describe Your Business Accurately Read guide → AI search visibility ### llms.txt Explained: An Optional Guide for Your Website Read guide → AI search visibility ### What Is AI Search Visibility? A Guide for Businesses Read guide →