: What is the default behavior of robots if there is no robots meta tag? Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots?

Posted in: #MetaRobots #RobotsTxt #WebCrawlers

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in:

<meta name="robots" content="index, follow" />

I'm asking because I want to know if I can safely remove the meta tag.

10.01% popularity Vote Up Vote Down

: Google analytics web data vastly different than the core reporting API data I am trying to pull the exact same data i am seeing in my dashboard via the core reporting API for Google Analytics.

@Debbie626

Posted in: #Api #GoogleAnalytics #Python

2 Comments

: Does putting Google Custom Search within your site affect traffic negatively? On the 26th of November we put Google Custom Search on our site. The day after and all days following traffic

@Debbie626

Posted in: #Google #GoogleCustomSearch #GoogleSearch #Seo

2 Comments

: Can a website require search engine robots to support cookies? Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple

@Debbie626

Posted in: #Cookie #RobotsTxt #SearchEngines #WebCrawlers

2 Comments

: How can I make my website appear in the top position of Google? I have added some of the keywords in a meta tag, but my site is still not in the top position when Googled. How can I achieve

@Debbie626

Posted in: #Seo

2 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@YK1175434

The default behaviour of robots if there is no robots.txt and robots meta tag is to follow links on the page and index it.

That's why <meta name="robots" content="index, follow" /> is not necessary if you want the default behaviour of robots for your page.

10% popularity Vote Up Vote Down

Feed

: What is the default behavior of robots if there is no robots meta tag? Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots?

More posts by @Debbie626

: Google analytics web data vastly different than the core reporting API data I am trying to pull the exact same data i am seeing in my dashboard via the core reporting API for Google Analytics.

: Does putting Google Custom Search within your site affect traffic negatively? On the 26th of November we put Google Custom Search on our site. The day after and all days following traffic

: Can a website require search engine robots to support cookies? Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple

: How can I make my website appear in the top position of Google? I have added some of the keywords in a meta tag, but my site is still not in the top position when Googled. How can I achieve

Login to post a comment!

1 Comments

Back to top | Use Dark Theme