: Homepage not being crawled by search engines My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache

@Mendez628

Posted in: #RobotsTxt #WebCrawlers #Wordpress

My website is using WordPress with the following robots.txt

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/cache
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*


I updated the robots.txt file 3 days ago and the problem I'm having is when I search for the site on Google or Bing it shows all sub pages but not the home page. Google says the following for the homepage:


A description for this result is not available because of this site's
robots.txt – learn more.


I tested using Google webmaster tools and it shows no errors.
It shows description of home page when is search with site:domainname.com.
Is this a WordPress issue or does my robots.txt file have errors.

Vote Up Vote Down

Login to post a comment!