: User-agent line needed in robots.txt? I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase. According to Google Webmasters

Posted in: #GoogleSearchConsole #RobotsTxt #Seo

I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase.

According to Google Webmasters Help section apparently you only need the following in your robots.txt file:

Disallow: /

But according to www.robotstxt.org/faq/prevent.html and other sources you have to write it like this:

User-agent: *
Disallow: /

I'm afraid that the bots still will index my site if I leave the User-agent: * part out.

So which is the proper way to do it?

10.02% popularity Vote Up Vote Down

: Splitting a Website in two sub domains by business activity Hello I am currently working on a website for a company that so far have been manufacturing yarns, however currently they started

@Carla537

Posted in: #Domains #Google #Seo #Subdomain

1 Comments

: Apache htaccess - modify query string with mod_rewrite I've recently changed scripting on my website, and this has resulted in a few 404 errors as some very old queries are no longer being

@Carla537

Posted in: #Apache #Htaccess #ModRewrite

1 Comments

: Is there anything unusual or nefarious about private IP addresses in server logs? A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227

@Carla537

Posted in: #AccessControl #Apache #WebCrawlers

1 Comments

: A social network called "socialnetwork" in Google Analytics I have stumbled across a social network that Google analytics recognises as simple "socialnetwork" which is referring some users to my

@Carla537

Posted in: #Analytics #GoogleAnalytics #Referrer #SocialNetworks

0 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Ogunnowo487

The original robots.txt specification is www.robotstxt.org/orig.html, where it says:

The record starts with one or more User-agent lines […]

So yes, User-agent is required.

10% popularity Vote Up Vote Down

@Gretchen104

As far as I know and as far as I have search for it, there is no robots.txt documentation saying you can drop the user-agent line. I would not rely on this.

10% popularity Vote Up Vote Down

Feed

: User-agent line needed in robots.txt? I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase. According to Google Webmasters

More posts by @Carla537

: Splitting a Website in two sub domains by business activity Hello I am currently working on a website for a company that so far have been manufacturing yarns, however currently they started

: Apache htaccess - modify query string with mod_rewrite I've recently changed scripting on my website, and this has resulted in a few 404 errors as some very old queries are no longer being

: Is there anything unusual or nefarious about private IP addresses in server logs? A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227

: A social network called "socialnetwork" in Google Analytics I have stumbled across a social network that Google analytics recognises as simple "socialnetwork" which is referring some users to my

Login to post a comment!

2 Comments

Back to top | Use Dark Theme