Thread: Robots.txt: Detailed Specifications Recently Published by Google

Hi Fellows,

I found this and thougt could be intersting for some of you.

With this, webmasters and SEOs have new reading material for this weekend. Google has published a very comprehensive and detailed Robots.txt Specifications, Robots meta tag and X-Robots-Tag HTTP header specifications and how to control crawling and indexing by GoogleBot.

There are two threads that covering this new document. One is at WebmasterWorld and the other is at Google Webmaster Help.

I hope you find this useful. If yes, don't hesitate to hit thanks button

I dont know how to use Robot.txt,can you post an full example