: Using robots.txt to deny access to MediaWiki special pages using substring matching I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated

@Berryessa370

Posted in: #Indexing #Mediawiki #RobotsTxt #WebCrawlers

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively scanned by search engines with queries like:

/index.php/Special:Whatlinkshere/some_topic
or as well
/index.php?title=Special:Whatlinkshere&target=some_topic
where some_topic is an article of the wiki.

These requests seem to be of very low benefit but they consume a good deal of bandwidth and in addition I fear those automatically generated pages are not so good for my site's quality reputation in the serchengines evaluation.

As the requests a mostly done by 'good' engines such as Google or Bing I am quite sure they would obey robots.txt. So I added the following robots.txt to the folder of the base url someurl.com (I added the whole robots.txt, even though only line 1 and 6 are relevant for the queries named above):

User-agent: *

Disallow: User:
Disallow: Discussion:
Disallow: MediaWiki:
Disallow: Special:

Disallow: /login.php
Disallow: /profile.php

Disallow: /author/
Disallow: /category/
Disallow: /tag/


This robots.txt is active for about two days now and has been crawled already, but still there are many requests to URLs like the above which I thought blocked.

So I have the following questions now:

1) is the above logic correct and capable of denying access (to well behaving bots). Especially I wonder whether Disallow: Spezial: correctly works as a wildcard to deny all request having "Special:" in URL or in parameter. I also wonder if the ":" in "Special:" might be a problem.

2) If so why then there is no effect yet? I consider I just have to grant more time to see the effect?

3) Will denying in the robots.txt lead to the de-indexing of this sites from the searchengines results? If not how can I get this huge amount of automatically generated URLs de-indexed?

Vote Up Vote Down

Login to post a comment!