: What is the correct way to write my "robots.txt" file? I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google

I have written the following code inside my robots.txt file:

User-Agent: Googlebot
Disallow:

User-agent: Mediapartners-Google
Disallow:

Sitemap: example.com/sitemap.xml

Is my robots.txt is correct? I only want two user agent to follow my site (i.e., Googlebot & Mediapartners-Google).

10.02% popularity Vote Up Vote Down

: Would changing IP addresses affect my SEO We're about to change our website from a shared hosting account to run on a VPS, mainly so we have our own resources and control over server settings.

@Gonzalez347

Posted in: #ChangeManagement #IpAddress #Seo

2 Comments

: Make Apache encode or replace quotes instead of escaping them? In the dcoumentation I read Format Notes For security reasons, starting with version 2.0.46, non-printable and other special characters

@Gonzalez347

Posted in: #ApacheLogFiles

1 Comments

: Google Analytics 'Visits' count sudden jump I have a site that I've been closely monitoring with Google Analytics. Each day traffic has been gradually increasing, and the #1 source is "Google/Organic".

@Gonzalez347

Posted in: #GoogleAnalytics

1 Comments

: Confused for writing robots.txt file I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt

@Gonzalez347

Posted in: #GoogleSearchConsole #RobotsTxt

2 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Nimeshi995

Almost, see the bottom of the section named, "Blocking user-agents" in the following: Google Webmaster Tools: Block or remove pages using a robots.txt file

According that, you should have:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Mediapartners-Google
Allow: /

sitemap: example.com/sitemap.xml

See the bottom of the page in the above link for how to test your robots.txt file.

10% popularity Vote Up Vote Down

@Tiffany637

Nearly, you need to Disallow all the other bots first though.

The wild card (*) below means all bots.

User-agent: *
Disallow: /

User-Agent: Googlebot
Disallow:

User-agent: Mediapartners-Google
Disallow:

Sitemap: example.com/sitemap.xml

Although please note, not all web crawlers will obey robots.txt.

10% popularity Vote Up Vote Down

Feed

: What is the correct way to write my "robots.txt" file? I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google

More posts by @Gonzalez347

: Would changing IP addresses affect my SEO We're about to change our website from a shared hosting account to run on a VPS, mainly so we have our own resources and control over server settings.

: Make Apache encode or replace quotes instead of escaping them? In the dcoumentation I read Format Notes For security reasons, starting with version 2.0.46, non-printable and other special characters

: Google Analytics 'Visits' count sudden jump I have a site that I've been closely monitoring with Google Analytics. Each day traffic has been gradually increasing, and the #1 source is "Google/Organic".

: Confused for writing robots.txt file I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt

Login to post a comment!

2 Comments

Back to top | Use Dark Theme