Hi Again,
May webmasters having problem in creating robots.txt to block unwanted crawler to crawl their website pages. It is now very easy to create a robotos.txt using Google. Webmaster wrote many posts regarding robots.txt in Google Forum and finally they added this new advanced feature in Google Webmaster Help.
It is very easy to create a robotos.txt file, to do so first create an account with Google Webmaster Tool and then submit your website. Once you submit your website they will ask for Verification, choose any verification methods for your website. Once they verify ownership of website then only you can create a robotos.txt.
Now go to Tools section from the left panel and there you will find Generate robotos.txt link. Then follow some simple steps
1. Choose your default access : Here you have to mention if you want to block all robots or want to allow all robots.
2. Specify any additional rules : This is an optional feature but if in first step you have selected Allow all robotos, and now you want to some robots then select robot from the combo box and then enter the file or directory path (e.g. /images/ or /temp/index.htm)
3. Then click on download and you have done.
It is very important to check you have written correct file that you want to block. To do so go to Analyze robots.txt, here you can see whether your robotos.txt file block specific urls.
Post a comment if you have any question.
Regards,
Bhavesh Goswami.
