: HTTP: How to be deleted from search engines at a certain point in time in the future? Is there a way to tell search engines, that a page they crawl should be included in the search results

@Mendez628

For google there is a meta tag called unavailable_after, which does exactly what I was looking for: It tells google to remove a certain page at a specific time in the future.

It is the only way to achieve what I was hoping to accomplish: Getting the pages removed automatically, at the right time, not relying on the crawler to come back and notice the 410 Gone response, which can take a weeks after the content has been removed.

Example:


<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 25-Aug-2007 15:00:00 EST" />


Or with HTTP header, for PDFs etc.:


X-Robots-Tag: unavailable_after: 23 Jul 2007 15:00:00 PST


Sources: googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html and www.google.com/support/webmasters/bin/answer.py?answer=79812
I could not find out if Bing, Yahoo & Co have adopted this Google specific tag.

Vote Up Vote Down

@Merenda212

First of all you don't have control what search engines crawl and what they put in their index.

BUT, Google for instance is taking your information about the live time of your pages very serious. So if you add the correct HTTP header it will consider those information. You can also add some information into your robots.txt about which pages are invalid.

There are also the Webmaster tools where you can tell Google to remove pages from the index.

On the official Google webmaster blog you will find very helpful information about removing URLs from the index and how to reinclude content. There they say you can remove URLs by:


using 410,
robots.txt or the
noindex meta tag

Vote Up Vote Down