Robots.txt Optimization

Robots.txt passes an instruction to search engines about which page the landing page should crawl, index and which page shouldn’t.