What is Custom Robot txts?

[priyaroy]

Guys what is the option of custom robots txt in wordpress and blogspot?

[swikriti.sharma]

Custom robots.txt is a way for you to instruct the search engine that you don't want it to crawl certain pages of your blog.Robots.txt is a file which tells the search engine crawlers what page to crawl and what not to.

[virginoilseom]

- Custom robots.txt is a way for you to instruct the search engine that you don't want it to crawl certain pages of your blog.“Crawl” means that crawlers, like Googlebot, go through your content, and index it so that other people can find it when they search for it.

[simon56]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[damponting44]

Robot.txt is a way to tell search engines whether they are allow to index a page in the search result or not. The bots are automatic, and before they could access your site, they check the Robot.txt file to make sure whether they are allowed to crawl this page or not.