Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.txt Generator tool allows you to generate robots.txt files for your website for free without signup in simple steps.

The generator can be uploaded to the root of your domain so that a crawler reading your website can see what rules you can set for your website and what pages you want and don't want to index.

Advantages Of Using Greatseotools Robots.txt Generator -

  1. Generate a robot.txt file for your website.
  2. Requires very few details.
  3. No signup required for use.
  4. Free to use no payment required.
  5. The robots.txt file can be generated unlimited times.

Why generate Robots.txt file for my website?

Obviously, there is our benefit if you make a file that is we get a user for our website but you are benefitted forever by all search engines because it helps them to easily locate the sitemap of the website.

Also, you can all those pages that should not be indexed by those search engine crawlers like the admin panel and some other main pages that you don't want to get indexed.

How to use the Robots.txt tool?

Step 1 - Check default value for all robots 

Step 2 - Choose Crawly Delay from all the options

Step 3 - You can either enter or leave if you don't have a sitemap for your website.

Step 4 - Change the value from default for any of the robots you want

Step 5 - Add the directories you want to restrict the robots to crawl from

Now, Click on the create robots.txt button, and you will get a robot.txt file generated for you just copy paste it into a new file onto you your website root and name it robot.txt after that you will have a file opening at https://yourdomain.com/robots.txt if you see this you have successfully made a robots.txt file.