XML Sitemap URL
User-Agent Directives (2)
User-agent:
Disallowed Paths:
User-agent:
Disallowed Paths:
robots.txt Output
# robots.txt generated by MiniToolDesk - 100% Client-Side # Documentation: https://developers.google.com/search/docs/crawling-indexing/robots/intro User-agent: * Disallow: /admin/ Disallow: /api/private/ Disallow: /tmp/ Allow: / User-agent: GPTBot Disallow: / Sitemap: https://minitooldesk.com/sitemap.xml
Pin this tool to your browser side panel
Access this utility anytime with Alt+M without switching tabs. 100% offline & free.
How to generate robots.txt
- 1Enter your XML sitemap URL.
- 2Configure user-agents and disallowed sensitive paths.
- 3Download robots.txt and place it at the root of your domain.
Search Engine Indexing
Guides web spiders to important content while shielding private admin routes.
Frequently Asked Questions
Enjoying Robots.txt Generator?Help your engineering peers discover private, ad-free tools!
MiniToolDesk is 100% free and client-side. Your support keeps the web ad-free.
Related Developer Utilities
Popular
Diff Checker
Compare text and code with instant side-by-side and inline visual difference highlighting
Launch tool→
Case Converter
Convert strings between camelCase, snake_case, kebab-case, PascalCase, and UPPERCASE
Launch tool→
Word & Character Counter
Inspect word counts, character lengths, paragraph totals, and estimated reading time
Launch tool→
Remove Duplicate Lines
Deduplicate text rows, sort alphabetically, and trim empty lines in seconds
Launch tool→