Robots.txt Generator

Build a robots.txt file with your sitemap, blocked paths and choices for AI crawlers.

robots.txt

This tool runs in your browser. Nothing you type is sent to us or anyone else.

How to use it

Choose which crawlers you want to address, add any paths to block, add your sitemap address and decide whether to allow AI crawlers. The generator builds a robots.txt file you can copy and upload to the root of your site (example.com/robots.txt).

What robots.txt does, and doesn’t do

Robots.txt asks crawlers not to visit certain paths. It does not remove pages from search results, and it does not protect private content: a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex robots meta tag instead and let crawlers see it. To protect private content, use a password.

AI crawlers

Several AI companies publish the names of their crawlers so site owners can allow or block them. Blocking them may keep your content out of their training data or their answers; allowing them makes your content available to their systems. Decide based on whether you want your pages used as a source in AI answers.

Before you upload

Never block CSS or JavaScript files that your pages need to display.
Don’t block pages you want to appear in search.
Test the file in Google Search Console after uploading, and check the Page indexing report for blocked pages over the following weeks.

Prefer an expert review?

Send us your site and we will reply with the first things worth fixing.

Request a review