: Does specifying full webpage path in robots.txt would affect my website? It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some

@Ann8826881

The path specified in the Disallow: field is simply a URL prefix. So, any URL that starts with this prefix will be blocked.

Disallow: /directory/wp-admin/path/label


From your example, this will therefore block all of the following URLs:

/directory/wp-admin/path/label
/directory/wp-admin/path/labelfoo
/directory/wp-admin/path/label/
/directory/wp-admin/path/label/bar.html


But will not block:

/directory/wp-admin/path/foo
/directory/wp-admin/path/
/directory/wp-admin/hello.html
:


Googlebot does not see the separate directories that make up the path. It is simply one value, one URL prefix.

More information on the Google Developers website: developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Vote Up Vote Down

@Jessie594

Robots.txt treats each path directive individually...

So, for example:-

User-agent:*
Disallow: /directory/wp-admin/path/label


This would disallow crawling of the directory label but not everything preceeding this directory nor following it.

User-agent:*
Disallow: /directory/wp-admin/path/label/*


This would disallow the crawling of everything within the directory label , including sub directories and their contents and the directory itself.

Vote Up Vote Down