PDA

View Full Version : Should You Block Google from Crawling Internal Search Result Pages?



Rscomp
06-23-2019, 08:33 PM
Should You Block Google from Crawling Internal Search Result Pages?

PrimeItSolution
06-23-2019, 09:21 PM
The Google Webmaster Guidelines still state that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.” ... However, blocking internal search pages in your robots.txt doesn't seem the right solution

imriakapoor
06-23-2019, 10:00 PM
Why would you want Google to be blocked from not crawling internal search result pages?

soniya_ss
06-23-2019, 10:04 PM
yes, you can block Google bots from crawling the subpages of your website using robots.txt.