A robots.txt file is that file in which you tell search engine robots or web crawlers to index only specific web pages that belong to your website or blog.
Updated : 4 May 2017
Why do you need a robots.txt file for Blogger?
1. Blogger archives all your posts monthly and yearly. This archive is treated as duplicate content by search engines like Google and thus your blog will have a lower ranking in Google's search engine results.
2. Specifying a sitemap to search engines
3. Not allowing search engines to crawl particular pages on a website like Login Page
4. Not allowing search engines to crawl some images and js scripts
For example have a look at this sample robots.txt file:
https://www.sundeepmachado.com/robots.txt
Note the below text:
Disallow: /search
What the above text tells Google is not to index web pages (blog posts) with the word search in its URL.
This effectively means that a web page having an address like the one below won't be indexed by a Google.
https://www.sundeepmachado.com/search/label/apps
How to create a robots.txt file?
A robots.txt file can be created easily using Google Webmaster Tools.
Please try the below steps with extreme caution. If you don't know what you are doing, please use the example robots.txt file and replace the text in it with your blog name.
Steps to create a robots.txt file
1. Go to Google Webmaster's Tools by clicking here: https://www.google.com/webmasters/tools/. Make sure your site is already added to Webmaster Tools. If not, visit this page: Google Webmasters for Blogger that helps you to add a blogger blog to Google Webmaster Tools
2. Click on Crawler access
3. Just copy the code in your Blogger template. Make sure you test the code with the help of Test button.
To know more
1. Robots.txt file tutorial
2. http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=164734&rd=1
