Robots.txt Generator & Live Tester

Easily generate standard or custom robots.txt files to control how search engines and AI scrapers crawl your site. Choose CMS presets (WordPress, E-Commerce), block AI bots, or validate any live website robots.txt.

Interactive Rule Builder WordPress & Store Presets Block AI Scrapers (GPTBot / Claude) Live URL Validator

Configure Directives

Website URL *

Used to automatically construct your Sitemap directive path.

Configuration Preset

Block AI Data Scrapers

Disallows GPTBot, ClaudeBot, CCBot, Google-Extended, and Bytespider from training on your content.

Crawl-Delay (Optional)

Additional Disallowed Paths (One per line)

Paths you want hidden from Google (e.g. /downloads/private/).

Sitemap XML URL

Generated `robots.txt`

Installation Step: Upload this downloaded file to the root directory of your website (e.g. public_html/robots.txt) so it becomes accessible at https://yourdomain.com/robots.txt.

Website Domain to Validate *

Enter your website URL (e.g. https://www.yourwebsite.com)

Try sample:

Robots.txt Best Practices & Directive Rules

User-agent

Specifies which crawler the rule applies to. User-agent: * targets all web robots, while Googlebot targets Google specifically.

Disallow vs Allow

Disallow: /admin/ hides backend portals. Be careful: Disallow: / accidentally tells Google not to index your entire website!

Sitemap Directive

Always include an absolute link to your XML sitemap at the bottom of robots.txt to assist search engines in discovering newly published pages.

Need a Professional Technical SEO Audit?

WebSupports optimizes crawl budgets, fixes indexing errors in Google Search Console, and resolves duplicate content issues.

Chat with SEO Specialist

Back to all tools

Robots.txt Generator & Live Tester