Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

A Robots.txt Generator is a tool used to create or generate the robots.txt file for a website. The robots.txt file is a text file placed in the root directory of a website that provides instructions to web crawlers or robots about which pages or sections of the site they are allowed to crawl and index. Here's how a Robots.txt Generator typically works:

• Website Input: Users input the URL of their website into the Robots.txt Generator tool.
• Pages and Directories Selection: The tool presents options for specifying which pages, directories, or sections of the website should be allowed or disallowed for crawling by search engine bots. Users can select individual pages or directories or use wildcard characters (*) to apply rules to multiple pages or directories simultaneously.
• User-Agent Selection: Users can specify directives for specific search engine crawlers or user-agents, such as Googlebot, Bingbot, or others. This allows users to customize crawling instructions for different bots if needed.
• Allow and Disallow Rules: Users can set rules to allow or disallow crawling of specific pages or directories by search engine bots. For example, "Disallow: /private/" would instruct bots not to crawl any pages within the "private" directory of the website.
• Sitemap Declaration: Some Robots.txt Generator tools offer the option to declare the location of the website's XML sitemap within the robots.txt file. This helps search engine bots discover and crawl important pages more efficiently.
• Advanced Options: Advanced users may have the option to add custom directives or advanced rules to the robots.txt file, such as crawl-delay directives to control the rate of crawling, or directives for specific bots or user-agents.
• Preview and Download: After configuring the robots.txt file settings, the tool generates the robots.txt file based on the specified rules and directives. Users can preview the generated robots.txt file and then download it to their computer.

Popular Robots.txt Generator tools include tools provided by SEO platforms like Yoast, SEMrush, and Ahrefs, as well as standalone online generators such as Robots.txt Generator by Small SEO Tools. These tools are essential for ensuring proper control over which pages search engine bots can access and crawl on a website, thereby helping to optimize crawling efficiency and manage SEO performance.