: Wordpress robots.txt is this reasonable？ this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms.

all the page urls like mydomain.com/it/page1, mydomain.com/en/page1 (for /it/ and /en/ just edit in /wp-admin/post.php?post=1&action=edit&message=1, create a Parent page - it or en, then save page under theme)

I stored photos in /thumb/ and /image/ folder. here is my full robots.txt.

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /thumb
Allow: /image

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

Sitemap: mydomain.com/sitemap.xml
User-agent: *
Disallow: /wp-login.php
Disallow: /wp-admin/
Disallow: /signup/
Disallow: /cgi-bin/
Disallow: /wp-
Disallow: /wp-admin/
Disallow: /signup/
Disallow: /cgi-bin/
Allow: /it/
Allow: /en/
Allow: /es/
Allow: /fr/
Allow: /de/

BTW, if i submit my site to www.google.com/webmasters/tools/robots-analysis-ac how many hours, google will crawl my robots.txt and sitemap.xml? and if I will apply for google adsense, must I first finish a seo? (google finish catch my robots.txt and sitemap.xml?) Many thanks.

10.01% popularity Vote Up Vote Down

: Mod_rewrite for directories One of our developers who built the framework that our website uses is no longer with us and I appear to have a stack of issues across the site currently, one being

@Harper822

Posted in: #Htaccess #ModRewrite

1 Comments

: What is the difference between doing a "link:http://www.example.com" google search and the "links to your site" in Webmaster Tools? I've tried to do two kinds of searches which I had previously

@Harper822

Posted in: #GoogleSearch #GoogleSearchConsole

1 Comments

: Alternative to nofollow: custom 302 url shortener? Here's the scenario: lots of blogging platforms make it tedious to insert nofollow into links within the post content. I.e., you need to edit

@Harper822

Posted in: #Http #Nofollow #Redirects #Seo

2 Comments

: Port forward based on external IP (for VPS hosting) What I want to do is to host a VPS. First, I'd like to set up a static IP address that forwards to my home IP address (so I can have

@Harper822

Posted in: #SharedHosting #Vps

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Jessie594

I would be cautious only allowing Google image bot to your image folders. We don't know if Google's user agent will simply be Google or how their search bots contribute to their image search results.

The latest version of WordPress also includes I believe index files for folders in case you have world readable folders.

How do you plan to handle your wp-content folders? I would suggest you simply let Google crawl your entire site, include your sitemap file in the robots if anything. And just run a site: search occasionally to see what has been indexed. But a properly configured server and htaccess file should prevent unwanted files from getting indexed.

Do you run any CGI scripts? If not just delete that folder you don't need it.

10% popularity Vote Up Vote Down

Feed

: Wordpress robots.txt is this reasonable？ this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page

More posts by @Harper822

: Mod_rewrite for directories One of our developers who built the framework that our website uses is no longer with us and I appear to have a stack of issues across the site currently, one being

: What is the difference between doing a "link:http://www.example.com" google search and the "links to your site" in Webmaster Tools? I've tried to do two kinds of searches which I had previously

: Alternative to nofollow: custom 302 url shortener? Here's the scenario: lots of blogging platforms make it tedious to insert nofollow into links within the post content. I.e., you need to edit

: Port forward based on external IP (for VPS hosting) What I want to do is to host a VPS. First, I'd like to set up a static IP address that forwards to my home IP address (so I can have

Login to post a comment!

1 Comments

Back to top | Use Dark Theme