: Google is ignoring my robots.txt Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to

@Annie201

No Allow code?
Try add

User-agent: *
Allow: /
Disallow: /images/
Disallow: /upload/
Disallow: /admin/

Vote Up Vote Down

@Angie530

While Google won’t crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site can appear in Google search results.

User-agent: *
Disallow:/images/

According to the above rule, all robots are denied access to the /images/ folder of a site. Additionally, a specific rule can be set to explicitly disallow access to all files within a folder:

Disallow:/images/*

Vote Up Vote Down

@Sarah324

Google isn't ignoring your robots.txt as robots.txt does not tell Google not to list that content in its search results. It tells Google not to crawl content. So Googlebot cannot find this content on its own. But if it does find this content it will still list it in the search results.

To keep content out of the search results you need to use the x-robots-tag HTTP header. It tells search engines not to index that content. If you're using Apache you can place a file called .htaccess in each of the mentioned directories with the following line in it:

X-Robots-Tag: noindex


That will prevent the content of those directories to not appear in the search results.

Vote Up Vote Down