: Can a website require search engine robots to support cookies? Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple

Posted in: #Cookie #RobotsTxt #SearchEngines #WebCrawlers

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other irrelevant crap.

Lately I have noticed that some web sites require cookies to be turned on, even if you are a bot. What is the purpose of this? Why would a search engine robot need to accept cookies? Or is this just because the guy who designed the web site screwed up?

10.02% popularity Vote Up Vote Down

: What is the default behavior of robots if there is no robots meta tag? Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots?

@Debbie626

Posted in: #MetaRobots #RobotsTxt #WebCrawlers

1 Comments

: Google analytics web data vastly different than the core reporting API data I am trying to pull the exact same data i am seeing in my dashboard via the core reporting API for Google Analytics.

@Debbie626

Posted in: #Api #GoogleAnalytics #Python

2 Comments

: How can I make my website appear in the top position of Google? I have added some of the keywords in a meta tag, but my site is still not in the top position when Googled. How can I achieve

@Debbie626

Posted in: #Seo

2 Comments

: Dealing with thousands of malicious broken links I have a problem similiar to this one - thousands of incoming links that go to nonexisting pages. The difference is that all of them include

@Debbie626

Posted in: #Redirects #Seo

1 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Eichhorn148

There are two other possibilities that spring to mind:

Websites can tell that you are pretending to be Googlebot. Google issued a procedure for verifying Googlebot that these websites might be using. They may be requiring cookies only because you are not really Googlebot
The sites don't care about having their pages included in the search indexes. When you don't care about being indexed, it doesn't really matter if you treat the search bots badly.

10% popularity Vote Up Vote Down

@Sarah324

Tracking comes to mind. Cookies would enable the site to ID a bot as it crawls the site and allow them to separate it from human traffic when analyzing browsing habits.

10% popularity Vote Up Vote Down

Feed

: Can a website require search engine robots to support cookies? Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple

More posts by @Debbie626

: What is the default behavior of robots if there is no robots meta tag? Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots?

: Google analytics web data vastly different than the core reporting API data I am trying to pull the exact same data i am seeing in my dashboard via the core reporting API for Google Analytics.

: How can I make my website appear in the top position of Google? I have added some of the keywords in a meta tag, but my site is still not in the top position when Googled. How can I achieve

: Dealing with thousands of malicious broken links I have a problem similiar to this one - thousands of incoming links that go to nonexisting pages. The difference is that all of them include

Login to post a comment!

2 Comments

Back to top | Use Dark Theme