What is robots.txt

Robots.txt file is an ASCII text file having specific instructions for engines robots about specific content that they are not allowed to index or to visit the pages.

Robots.txt is by no means mandatory for search engines, but normally search engines obey what they are asked not to perform.

    Ex: If a website has two different pages, one of them would be for viewing and other would certainly for printing. You need to exclude printing version from the crawling otherwise that may result you in duplicate penalty. In this case, you can make the printing page in the robot.txt file, so that spiders cannot crawl that page.

This doesn't hurt, BUT if you are using Google AdSense on any of your pages, then Google requires using a special User-Agent: type to be used in the robots.txt file.

    Note: Please don't try to spam in that .txt file by Keyword Stuffing, since robots only read that text file to find out their rights to, which folders and pages they are allowed to index.

You can also ban your own site by having a robots.txt with the wrong code.

The following are the two examples of robots.txt code are below.

This example allows all robots to visit all files because the wildcard "*" specifies all robots.
          User-agent: *
          Disallow:

This example keeps all robots out:
          User-agent: *
          Disallow: /

Download This Tutorial in to PDF: