robots.txt?

[Mike]

Hi there,

Would someone please tell me what robots.txt is? I have heard it loads of times but never actually come across it on my server...

Thanks,
Mike

[Chris]

check out http://www.robotstxt.org

[Mike]

Ok, thanks Chris.

Just a couple of questions I have...

Would you use a robots.txt file or the Robots meta tag? and, If I am doing a redirect thing to link to other sites should I use this robots technique so that I make sure it count as a external link?

Thanks for information so far,
Mike

[Chris]

robots.txt is better than the robots meta tag

[Mike]

Thanks.

[incka]

Looks interesting... Im going to use it to block my stats... I seem to be looking pagerank because google is following all the links in it.

[s2kinteg916]

bringing back the old post ... if you make a robots.txt file do u need to linclude it in a webpage via metatags ?

[lo0ol]

Just place robots.txt on your domain and you're set to go. The search engines and most other crawlers/spiders will check for it there. If you would like to view examples of other sites' robots.txt, just check them out yourself at theirdomain.com/robots.txt.

[incka]

Google has a page indexed even though my robots.txt tells it not to, but it had indexed it before I told it not to, but it should delete its index of it shouldn't it...

[s2kinteg916]

i believe you will have to wait till google updates...i seem to be in the same boat...

[incka]

The problem is I've had that robots.txt file there since September and it hasn't found it...

[r2d2]

I doubt it would delete the index of the file, robots.txt will just stop the next googlebot from looking at the file I would think.