: Robots are still disallowed even without a robots.txt file I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I

@Jamie184

To combine everything into one big answer, here's what you should do...


Make sure your robots.txt is correct. Here's what it should look like if you want the crawlers to index everything on your site:

User-agent: *
Disallow:

Please note that the Allow field is not officially supported by all crawlers (Disallow is the universally accepted field).

Create an XML Sitemap that lists the pages on your site. You can do this manually, or you can use an automatic generator.

Register your site with Google Webmaster Tools.

Submit your XML Sitemap to Google Webmaster Tools.



Once you've completed these steps, your site will be well on its way to being indexed.

Vote Up Vote Down

@Radia820

So the robots file still appears to be present but the code has changed since i looked at it first thing it now shows:-

User-agent: *
Allow: /


But to allow all, it should be (what it was this morning)

User-agent: *
Disallow:


You can find some more examples in the wiki article on Robots Exclusion Standard.

Could it be that one of your wordpress plugins is auto generating a robots file?

Vote Up Vote Down

@Angie530

So far, nothing seems immediately wrong. Instead, some of your assumptions appear to be wrong.

First, questions.
When you say: "but it [] shows that robots are disallowed" what is the "it" being referred to?

How long has it been since you deleted the robots file? That's not going to make any difference until you get crawled again.

Now, your search example suggests that you have been indexed, at least to some degree. But that doesn't mean the engines will decide all your pages will be worth actually returning as search results.

Beyond that, searches with the site: operator do not necessarily return everything indexed for a site, but only a selection. If you need to actually know how well your site's been crawled, you get that information from Webmaster Tools.

Vote Up Vote Down