Mobile app version of vmapp.org
Login or Join
Murray432

: Googlebot received Access Denied when crawling a directory I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google

@Murray432

Posted in: #403Forbidden #Googlebot

I'm facing the same problem with Googlebot not able to crawl my page. Mine problem doesn't have anything to do with www or not. Google can crawl all my pages except one directory and this directory contains all images, no code. (I have other images directory and Googlebot can crawl fine).

When checking Google Webmaster Tools for Crawl Error, the directory which Googlebot couldn't crawl and yield "Access Denied" error is: www.artistviewpoint.com/picture-to-painting/pictures/samples/.
Reading the guide from Google, I have already checked my robots.txt file and there is no log-in authentication of any sort. The only remaining tips which Google provide is to check the hosting provider or authentication using proxy which I do not know how to check.

10.01% popularity Vote Up Vote Down


Login to follow query

More posts by @Murray432

1 Comments

Sorted by latest first Latest Oldest Best

 

@Heady270

Its not just Googlebot that can't visit that page. If I visit in my browser I get a blank screen. If I try accessing it with curl on the command line I get:

$ curl --head www.artistviewpoint.com/picture-to-painting/pictures/samples/ HTTP/1.1 403 Forbidden
Content-Length: 0
Server: Microsoft-IIS/8.0
X-Powered-By: ASP.NET
Date: Sat, 30 Nov 2013 11:28:25 GMT


Google is not saying that they can't access any of the images in this directory. They are saying that they cannot view an index page for this directory. This is not really a problem. Configuring your server to deny access to a list of all your images is a very common setup. The error isn't going to prevent Googlebot from accessing, indexing, or ranking the rest of your site.

If you really want to get rid of the error you could:


put an index.html file in that directory
configure your server to redirect a request for an index.html file in that directory to your home page.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme