: How to block certain URL from robots.txt file? This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/

This is simple question but also I have a confusion.

Below is the code which I wrote in my robots.txt:

User-Agent: *
Disallow: /phpmyadmin/
Disallow: /login/
Disallow: /newsletter/
Disallow: /articlesubmission/
Allow: /

Sitemap: www.example.com/sitemap.xml

I want all bots to follow my site but they should not follow these URLs:
www.example.com/phpmyadmin/..... http://www.example.com/login/..... www.example.com/newsletter/..... www.example.com/articlesubmission/.....

My doubt: Is my above robots.txt code is correct?
I followed this code from StackOverflow robots.txt.

10.01% popularity Vote Up Vote Down

: Is my .htaccess rule creating an infinite loop? Whenever I type a made up address that includes a directory index it removes it. The problem is when the the address includes a forbidden subdirectory.

@Margaret670

Posted in: #Apache #Htaccess #ModRewrite

0 Comments

: Using Google Tag Manager to define the page type So, I am looking to add a tag that I want to use for A/B testing, however we don't have a page-type URL structure. Fortunately the tool can

@Margaret670

Posted in: #Javascript #TagManager

1 Comments

: Sitemap submission : Do sitemaps need to be resubmitted when they are updated? In short: Should I resubmit my sitemap through Google Webmasters Tools every time I replace a sitemap.xml with

@Margaret670

Posted in: #Content #Seo #Sitemap #XmlSitemap

3 Comments

: Looking up IP address of an area, town or State How can I find the IP of a location? I know I can lookup an IP's physical location anywhere but what about looking up the IP's of a specific

@Margaret670

Posted in: #Geolocation #Geotargeting #IpAddress

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Eichhorn148

Yes, you have defined your entries correctly.

Just to be clear, adding the name of a directory with a trailing slash will block that directory and everything within it; including if there is a live page on the directory address.

However your Allow: / is superfluous and will have no effect as you are already allowing the entire site apart from the directories you have disallowed, so this entry should be removed.

You can find more info on how robots.txt works with Google here: Robots.txt Specifications

10% popularity Vote Up Vote Down

Feed

: How to block certain URL from robots.txt file? This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/

More posts by @Margaret670

: Is my .htaccess rule creating an infinite loop? Whenever I type a made up address that includes a directory index it removes it. The problem is when the the address includes a forbidden subdirectory.

: Using Google Tag Manager to define the page type So, I am looking to add a tag that I want to use for A/B testing, however we don't have a page-type URL structure. Fortunately the tool can

: Sitemap submission : Do sitemaps need to be resubmitted when they are updated? In short: Should I resubmit my sitemap through Google Webmasters Tools every time I replace a sitemap.xml with

: Looking up IP address of an area, town or State How can I find the IP of a location? I know I can lookup an IP's physical location anywhere but what about looking up the IP's of a specific

Login to post a comment!

1 Comments

Back to top | Use Dark Theme