Tiny Toolkit

Robots.txt Generator

Build a robots.txt file visually by adding user-agent rules, Allow and Disallow paths, an optional crawl-delay, and a sitemap URL — then download the finished file. The output updates in real time so you can see the effect of each rule as you add it. Useful when setting up a new site or refining crawler access without memorising the robots.txt syntax.

All processing happens in your browser. No data is sent to any server.

Frequently Asked Questions

What is a robots.txt file?
robots.txt is a plain-text file at the root of your website that instructs web crawlers which pages or directories they may or may not access.
Does Disallow prevent a page from being indexed?
Disallow blocks crawlers from fetching the page, but if other sites link to it, search engines may still index the URL without content. Use a noindex meta tag to fully prevent indexing.
Where should the robots.txt file be placed?
It must be at the root of your domain; for example, https://example.com/robots.txt.

Related Tools

Hash Verifier — Check SHA-256, MD5 & More

Verify a file's integrity by checking its hash against an expected checksum. Supports SHA-256, SHA-1, MD5, and more. Runs in your browser — nothing uploaded. Free.

JWT Decoder

Paste a JSON Web Token to decode and inspect its header, payload, and expiry — entirely in your browser.

Regex Tester

Test regular expressions against a string with live match highlighting, flag toggles, and a match details table.