: Robots.txt not updated I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion

@Sarah324

May be that link is in indexed database of Google. Don't worry when it has been removed from Google database it will not show in SERPs.

Vote Up Vote Down

@Ravi8258870

Robots.txt can only prevent a search engine from crawling, it won't prevent indexing if the page or file has links from pages, internal or external to your site, which are crawlable.

To be absolutely sure that a page won't be indexed, apply <meta name="robots" content="noindex">. For any non-HTML content, like images of PDFs, use the equivalent HTTP header: X-Robots-Tag: noindex.

If you use these methods, it's unnecessary to submit requests for Google to remove them for you.

More discussion of both from Google here.

Vote Up Vote Down

@Goswami781

robots.txt only specifies how Google should or should not crawl your site and as Mike said if you solely rely on this vector it will take a long time to have the pages removed from the index.

So you will also need to make a removal request if you want it to happen faster and have Google remove the pages from the index. When you are in Webmaster Tools go to Optimization > Remove URLs and follow the instructions.

Vote Up Vote Down

@Shelton105

It takes some time for Googlebot to crawl your site. This amount of time can vary depending on the number of links to your site, among other factors. After your site is crawled, the URLs in the robot.txt file will be updated.

Vote Up Vote Down