: What is a good robots.txt for WP? What is the "best" setup for robots.txt? I'm using the following permalink structure in Wordpress: /%category%/%postname%/. My robots.txt currently looks like this

@Tiffany637

The WordPress Codex has an example SEO optimized robots.txt file: codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization
I use the following in the functions.php file of my theme to handle it:

// add to robots.txt
add_action('do_robots', 'roots_robots');

function roots_robots() {
echo "Disallow: /cgi-binn";
echo "Disallow: /wp-adminn";
echo "Disallow: /wp-includesn";
echo "Disallow: /wp-content/pluginsn";
echo "Disallow: /wp-content/cachen";
echo "Disallow: /wp-content/themesn";
echo "Disallow: /trackbackn";
echo "Disallow: /feedn";
echo "Disallow: /commentsn";
echo "Disallow: /category/*/*n";
echo "Disallow: */trackbackn";
echo "Disallow: */feedn";
echo "Disallow: */commentsn";
echo "Disallow: /*?*n";
echo "Disallow: /*?n";
echo "Allow: /wp-content/uploads";
}

Vote Up Vote Down

@Lee4591628

You should be safe to remove the */comments line if you want to index comments.
From what I've experienced, categories are definitely good to index. I would not disallow them.
Google should handle it. Check to see if pages you know could be duplicates have the "canonical" tag on them, which identifies the page as a unique resource. If you have duplicate content on these pages you could configure the plugin "All in one SEO pack" to probably take it off those pages: wordpress.org/extend/plugins/all-in-one-seo-pack/

Vote Up Vote Down