Parse, validate, and test your robots.txt against URLs and user-agents.
User-agent: * # all bots Disallow: /admin/ # block path Allow: /admin/public/ # exception Crawl-delay: 10 # seconds between requests Sitemap: https://example.com/sitemap.xml
Paste your robots.txt or click "Load sample" to start. Click Parse to see the rules in a structured table and review any warnings about syntax issues. Use the URL tester to verify whether a particular path is allowed for a given user-agent. Use the Wizard to generate a fresh robots.txt from a few simple choices.