: How can you tell google to stop indexing a resource? We have an app we are migrating. The old website had deeply nested robots.txt files at some point and then the old developers started handling
We have an app we are migrating. The old website had deeply nested robots.txt files at some point and then the old developers started handling those request with redirects to the root page. So:
a request to http:://example.com/foo/bar/robots.txt is now a 302 to example.com
I assume that because the old devs are doing a 302 instead of a 301, it continues to index that old path. Will a 301 to example.com/robots.txt tell google to abandon the old requests? What other things can I do to encourage these old paths to disappear from search engine indexing?
More posts by @LarsenBagley505
3 Comments
Sorted by latest first Latest Oldest Best
If you want google to stop indexing a site, page or directory and you don't want to redirect to a new site, you have a few options.
Delete all content
Add <meta name="robots" content="noindex,nofollow"> flags to your content.
Use webmaster tools to remove URLs from Google's index
Add a server-level password to your directories you no longer want indexed, via .htaccess file on Apache.
Remove the robots.txt files and have the urls simply return 410. It tells google the resource is gone and is never coming back.
No need to get complicated with robots tags.
robots.txt does not stop Google from indexing content. It only tells them not to crawl it. Content blocked by robots.txt can still be listed in Google's search results.
To actually prevent Google from indexing content you need to use the x-robots-tag HTTP header. You can do this in code or in your .htaccess file. But any pages you want to block from indexing should output the following HTTP header:
X-Robots-Tag: noindex
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.