: How to tell Googlebot to crawl just URLs ending in .html I have a big problem in Google Webmaster Tools. The number of 404 pages in the error report increases so fast that now I have over

@Eichhorn148

When Googlebot is crawling such a large number of bad URLs, it is almost always because your site is misconfigured and you are linking to the URLs incorrectly somewhere.

In your case it is the "show all articles" link. For example on this page I see the following in the HTML source code:

<a href="http://ermagazin.com/zakopao-zivu-djevojku-8-mjeseci-zbog-vjerovanja-da-ce-to-donijeti-bogatstvo-tanzanija-u-soku" class="more-articles-button">show all articles</a>

It appears that when I click on that button in a browser, I don't get to the 404 page. You must have some JavaScript that intercepts the click and causes browsers to to something else. However, Googlebot scans the HTML source code and finds that link. When it tries to follow it, it gets a 404 version of each and every article on your site.

You need to fix that link, and look for others like it.



Another thing that you can do is redirect requests for the URLs without .html to the correct versions. Since you are using WordPress, you might want to use a WorpPress 404 plugin that allows you to monitor and redirect 404 errors. I used to use one called "True Google 404" that ran the words in not found URLs through site search and automatically redirect to the proper page. Sadly that plugin appears not to be available anymore. I did a quick search but I didn't find any plugins that allow redirects based on patterns from WordPress 404s.

Vote Up Vote Down