Making A Robots.txt File

[thentavius]

I was wondering if any of you have any good information on making a robots.txt file. I have not created one yet, for any of our sites, and would like some pointers?

[mandy]

[quote author=thentavius link=topic=180.msg712#msg712 date=1123969358]
I was wondering if any of you have any good information on making a robots.txt file. I have not created one yet, for any of our sites, and would like some pointers?


[/quote]

robots.txt tutorial:  http://www.searchengineworld.com/robots/...torial.htm
robots.txt FAQ, list of robots: http://www.robotstxt.org/wc/robots.html
robots.txt file validator: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

[thentavius]

Thank you very much, Mandy!