: Stop Google crawling links completely I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow

@Heady270

Using Disallow: in robots.txt will prevent Googlebot from crawling the URLs at all. In fact, robots.txt is more of a tool for controlling your crawl budget than for controlling which pages are indexed. In some (albeit fairly rare) cases, Google chooses to included pages in the index even though they are in robots.txt based on the number of inbound links to those pages and the anchor text of those links.

So if you have a section of the site that you don't want to have Google crawling and using up your server resources create a robots.txt file like this:

User-agent: *
Disallow: /folder1/


Here is Google's full documention for robots.txt. They even support wildcards.

AJAX Checkboxes

I would not rely on javascript to prevent Googlebot from crawling anything. Googlebot now executes some javascript and its ability to do so is likely to improve in the future.

Hashbang #! URLs

Hash bang urls are specifically designed to be crawlable. When Googlebot encounters #! in a URL it makes a request to your site. Using a #! is also going to use up your crawl budget.

Using just a hash without the bang (#) in your URLs might work for you. Googlebot doesn't make requests to your server without the !.

Vote Up Vote Down