Robots.txt Is actually a file that exists on the basis directory of every Internet site and can be used to instruct engines like google on which directories/data files of the web site they could crawl and include things like in their index. Each time you use a internet search engine, https://news.rafeeg.ae