: How my robots.txt should look like for a single page app I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described

@Chiappetta492

I have found a way to do exactly what I want. It is nicely documented by google:


When your site adopts the AJAX crawling scheme, the Google crawler
will crawl every hash fragment URL it encounters. If you have hash
fragment URLs that should not be crawled, we suggest that you add a
regular expression directive to your robots.txt file. For example, you
can use a convention in your hash fragments that should not be crawled
and then exclude all URLs that match it in your robots.txt file.
Suppose all your non-indexable states are of the form
' #DONOTCRAWLmyfragment . Then you could prevent Googlebot from crawling these pages by adding the following to your robots.txt:

Disallow: /*_escaped_fragment_=DONOTCRAWL

Vote Up Vote Down