: Is it google webmaster crawler's fault or a virus? Last day when i visited google webmaster tools, i found crawler errors for 42 not found pages. all the pages were having "luminous-solutions.net"

@Yeniel560

A while ago I saw quite a few spammers taking advantage of poorly coded site searches. Some searches if they have nothing to return from their site will then show web results (IE Google search results). If they site doesn't keep their search results from being indexed a spammer can easily generate search results they know will return their site and build a few links to make sure the page gets indexed and voila they have links from your site to theirs.

I haven't seen this lately because Google doesn't like to show search results in their results and URLs from site search are pretty easy to pick out since they usually contain parameters like query= q= s= or search= so they do a good job of keeping them out of their index.

This is pure theory but along those lines it seems like it would be possible to exploit custom 404 pages (and more difficult for Google to detect). Many people create custom 404s to try and direct users to where they want to go, I've seen a bunch that appear to auto populate site search results based on URL parameters. I've also seen a bunch of custom 404s get indexed because they return a 200 status rather then true 404. It's logical if you combine the two you'd end up with a regular looking URL with a list of links to said spam site.

Regardless it looks as though your site is handling it properly.

Vote Up Vote Down

@Reiling115

Don't worry about them.

There are many spammy sites that create links such as that. GoogleBot honours them and tries to follow the URI only to hit a 404. If you do a search for link:http://article-stack.com/2010/08/page/luminous-solutions.net in respective search engines you'll often find the culprit (otherwise they'll have been taken down already).

A search for just link:luminous-solutions.net reveals a set of typical spam.

If you definitely do not want crawlers to follow them then create a deny all by default robots.txt and only allow the URLs you know will result in correct pages being displayed (but that may be considered overkill).

Vote Up Vote Down

@Ogunnowo487

Crawling your website with xenu does not show such links. Maybe the problem is in the submitted site map, can you send us a link to the sitemap?

Vote Up Vote Down