: Using asteriks in robots.txt Can spiders crawl the sites if there are no asterisks used on the robots.txt? For example: User-agent: * Disallow: .*/ping/.* then User-agent: * Disallow: /ping/ Please

@Odierno851

As @w3dk said you can't use dot asterisk (.*) in robots.txt with directory, it will probably gives you an error or you will not going to accomplish that you want.

.dot prefix used to block specific kind of file, like this.

User-agent: *
Disallow: *.pdf


This will block all pdf files to crawl in your website. No matter it is in root level or in sub folder.

Your second robots.txt clarification.

User-agent: *
Disallow: /ping/


Here all polite bots including Googlebot, MSNBot, Yandexbot allowed to crawl your all directory except /ping/ directory, but your /ping/ directory should be placed in root level, otherwise it won't work.

First example - example.com/ping/
Second example (This won't work) - example.com/sub/ping/

Vote Up Vote Down