: Is this a good WordPress robots.txt configuration? I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives

@Bethany197

Posted in: #RobotsTxt #Seo #WebCrawlers #Wordpress

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know that some search engines don't read wildcards.

sitemap: www.example.com/sitemap.xml
User-agent: *
Disallow: */comments/
Disallow: */feed/
Disallow: */trackback/
Disallow: *?replytocom
Disallow: /*.css$
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.php$
Disallow: /*/*.inc$
Disallow: /*/*.js$
Disallow: /*/*?s=*
Disallow: /*/cgi-bin/*
Disallow: /*/feed/
Disallow: /*/feed/*
Disallow: /*/trackback/
Disallow: /*/trackback/*
Disallow: /*/wp-*
Disallow: /*/xmlrpc.php
Disallow: /*?
disallow: /*?*
Disallow: /archives/
Disallow: /author
Disallow: /cgi-bin/
Disallow: /comments/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /index.php
Disallow: /login/
Disallow: /wp-*
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
Disallow: /wp-rss.php
Disallow: /wp-rss2.php
Disallow: /xmlrpc.php
Disallow: ?wptheme=
Disallow: /readme.html
Disallow: /license.txt
Allow: /wp-content/uploads

User-agent: Mediapartners-Google*
Disallow:
Allow: /

User-agent: Googlebot-Image
Disallow:
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Disallow:
Allow: /

User-agent: Googlebot-Mobile
Disallow:
Allow: /

User-agent: duggmirror
Disallow: /

User-agent: ia_archiver
Disallow: /

Vote Up Vote Down

Login to post a comment!