: Google is not indexing my site although it is crawling it I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt,
I have a site that has around 12,000 pages. A couple of days ago I uploaded the site, I removed the disallow option from my robots.txt, I run "fetch as Google" and I got the message:
URL and linked pages submitted to index.
The problem is that my site is not indexed yet although Google bot has started crawling it. When I search Google for my domain name I get:
A description for this result is not available because of this site's robots.txt
I have removed the disallow option and in fact robots.txt is as follows for the last 2 days:
User-agent: Googlebot-Image
Disallow: /
In fact, when I test the robots.txt file via the Webmaster Tools I get:
Allowed
Detected as a directory; specific files may have different restrictions
Google is crawling it at a slow pace (~400 pages per day).
How can I can tell Google to index my site before finishing with the crawling process? If this is not possible, can I speed up the crawl?
More posts by @Samaraweera270
1 Comments
Sorted by latest first Latest Oldest Best
I have removed the disallow option and in fact robots.txt is empty for the last 2 days.
If your robots.txt files goes away or becomes empty, Google sometimes assumes that this is an error and won't honor the change for some time. Instead of an empty robots.txt file, you should make one that explicitly allows everything:
User-agent: *
Disallow:
After you have changed your robots.txt file, Google usually takes at least a week and sometimes as much as a month to index a website. You'll just have to wait it out.
If it hasn't crawled the entire website, it will index the pages that it has crawled before completing a crawl the entire site.
The only way to speed up crawl process is to increase the reputation of the site with more high quality inbound links. The number of pages that Googlebot crawls and the rate it which it returns to re-crawl them are governed mainly by Pagerank.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.