Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

 

The Robots.txt Generator is a tool that creates a robots.txt file for your website. In other terms, this generator is certainly being useful to site owners in making their site Googlebot friendly. Also, you can choose which specific bot you want to have access to your website’s index and restrict other bots from doing the same thing. Similarly, you can be able to refine the crawling activities by specifying which bots should access certain files in your root directory and which bot should access another file.

How To Make a ROBOTS.TXT?

The file works similar to a sitemap that tells the robots which portion to crawl and which not.

Simply, use our Robots.txt Generator to generate robots.txt code for your website and upload the file to your site’s root directory. Remember that the robots.txt file should be accessible from “http://www.yourdomain.com/robots.txt”.

Robots File Looks Like:

A general or said to be as ‘default’ robots.txt looks like:

User-agent: *

Disallow:

But, don’t fret, you can readily create an advance file by using our smart robots.txt generator.

Advantages of Robots.txt:

Robots.txt is an account to give instructions to the search engine for telling them not to visit pages reciting in the file.

So, this instruction is useful for:

  • If you aim search engine to ignore any duplicate pages on your website

  • If you don’t want that the search engines index your internal search results pages

  • Prevent spamming

  • Simple, it helps to ban the bot crawlers of search engines from not entering to private content or folders

How to Use Robots.txt Generator?

Just stick to the following:

  • First thing, all the fields that you provided in the above tool are not mandatory, but you ought to select carefully. The first row has default values for all bots, and if you aim to keep a crawl-delay, then you ought to leave them as they are

  • Then, you see a field about sitemap, yes, you have to ensure that you have your sitemap and don’t forget to mention in the robot’s txt file

  • Right after, there are multiple options that you have to choose for search engines if you want search engines bots to crawl or not

  • The last field is for disallowing, where you will be able to restrict the crawlers from indexing the areas of the page

  • Finally, hit the ‘create and save as Robots.txt’ button, the robots.txt generator generates the file within seconds, seotoolsbot provides more 100 plus free SEO tools like backlink checker and backlink maker you should also check those as well.