View Full Version : robots.txt question
I'm sure I could figure this out on my own, I just wanted to be 5000% sure I do this right.
How would my robots.txt file look if I wanted to block .php extensions from being crawled by all search engines? Only .php, everything else is allowed.
Thanks!!
Best.Flash
11-04-2003, 04:08 AM
I dont think robots.txt files where designed to be that sophisticated.
I take it there's to many different php files to list them individually in robots.txt?
GCT13
11-04-2003, 08:44 AM
You could put all your .php files in a separate directory and then disallow that directory in robots.txt.
Is there no way to just block *.php?
GCT13
11-04-2003, 01:05 PM
Just found this (scroll to bottom):
http://www.searchengineworld.com/misc/robots_txt_crawl.htm
It indicates Google will recognize disallowing of certain file extensions, but other bots may not.
Thank you.. that's extremely helpful!
Chris
11-04-2003, 03:13 PM
You could put the robots meta tag on each php page.
Powered by vBulletin® Version 4.2.2 Copyright © 2024 vBulletin Solutions, Inc. All rights reserved.