Mobile app version of vmapp.org
Login or Join
Shelton105

: Does google crawl/follow link on a page with meta robots noindex We have added following meta tag on some of our pages: <meta content="noindex" name="robots"/> Does google crawl the content

@Shelton105

Posted in: #Google #Googlebot #MetaRobots #Seo

We have added following meta tag on some of our pages:

<meta content="noindex" name="robots"/>


Does google crawl the content and links on this page and just don't show it on the google search results.
If yes then how can i stop bot not to crawl the content and the links on this page ?

10.03% popularity Vote Up Vote Down


Login to follow query

More posts by @Shelton105

3 Comments

Sorted by latest first Latest Oldest Best

 

@LarsenBagley505

By default Google or any other modern search engine will crawl, index and follow the links in a page when the above meta tag is absent.

In your example Google will find the noindex instruction and this is what is expected to happen:


Google bot will crawl the page (if robots txt allows it)
Your page will not get indexed and will be removed from SERP, eventually.
The search engine will follow all the links in the page.
linked pages will get indexed.


Add nofollow

Add this instruction to the meta tag, this will tell Google not to follow the links. But, if these pages are linked from other pages without this instruction the pages will be indexed.

Do not allow Google to crawl the page

Make sure to Disallow o block access via robots txt. However, do this after Google has crawled the page and is aware of the noindex, nofollow instruccions.

10% popularity Vote Up Vote Down


 

@Phylliss660

Once a page is out there, Google is just going to crawl it, suck it up, and retain it in its vast and very long term memory.

If the page you want removed is in Google's SERPs, then the noindex alone won't resolve this, it'll simply advise new search engines not to rank it. You need to add nofollow,noarchive to that page's robots command (it can all be in one command) to flush it out of Google etc search cache, and stop the crawler following the links on that page.

Make sure you're allowing search engines to crawl that page via your robots.txt file, otherwise they'll never see this request.

Once your page has gone from the SERPs (do a few site:[yourdomain] queries in Google search box to check), then you can use the robots.txt to advise search not to go there. I'd opt to also keep the noindex,nofollow,noarchive page-level command in the page too, to keep it out of SERPs just in case the robots.txt file gets an edit.

It can take a while to fix this, but it'll get there.

10% popularity Vote Up Vote Down


 

@Voss4911412

The tag tells the search engines not to include the page in their listings. As far as i'm aware you can't stop a search engine crawling your page as it will need to craw the page in order to see if you remove the tag. (If you change your mind and want the page to be listed)

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme