: Prevent crawling of mobile site I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site

@Correia994

I think you should handle this issue differently. You want google to understand it's an alternate page version that is for different screen sizes. He will make sure a desktop user will be directed to the right place (as well as mobile users).

I would recommend the following:

In all /mobile/ pages use:

<link rel="canonical" href="/non-mobile-url"/>


And in the non mobile url put this

<link rel="alternate" media="only screen and (max-width: 640px)" href="/mobile/..." />

Vote Up Vote Down

@Nimeshi995

Each bot requires its own line.

Your robots.txt should look like this

User-agent: *
Allow: /

User-agent: Googlebot
Disallow: /mobile/
Allow: /

User-agent: bingbot
Disallow: /mobile/
Allow: /

User-agent: Googlebot-Mobile
Disallow: /
Allow: /mobile/

User-agent: bingbot-mobile
Disallow: /
Allow: /mobile/


Additional its always best to use Noindex and Robots.txt

You should consider using both robots and noindex... Simply use the following metas.

On the mobile site and not the normal site

<meta name="Googlebot" content="nofollow, follow">
<meta name="bingbot" content="noindex, follow">


On all the normal pages and not the mobile site use:

<meta name="Googlebot-Mobile" content="nofollow, follow">
<meta name="bingbot-mobile" content="noindex, follow">

Vote Up Vote Down