: Proper content for robots.txt on server maintenance page? I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index

@LarsenBagley505

robots.txt is meant to prevent search engines from accessing pages but for server maintenance pages, if the maintenance time is short (such as less than a couple hours), then I wouldn't worry about changing robots.txt and I'd just continue with the maintenance.

Search engine companies are only interested in indexing URLs that return a status 200 code (meaning page is good and has content people want to see). If your pages keeps returning status 503, then chances are they will eventually no longer be indexed.

I would suggest only forcing a noindex on pages that never offer value to the public.

Vote Up Vote Down

@Cugini213

you set the 503 page with your server configuration (Apache → htaccess), like

RewriteRule .* example.com/maintenance.html [R=503,L]


or

ErrorDocument 503 /maintenance.html


The page example.com/maintenance.html should be set to noindex, because in other case it appears in index. But it should remain crawlable - no crawling exclusion with

disallow: /maintenance.html,

because, in this case Google could not read the noindex rule.

Vote Up Vote Down