: How to Crawl a website requires cookies for audit? Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is:
Situation: My Client's website requires cookies to access it.
Users should choose (Language and country) to access the website.
The problem is: Whenever I try to crawl the website using any software (DeepCrawl or Screaming Frog), the crawler keeps getting the same page that asks for Language and country for each URL.
Question: How to allow the crawler to bypass, or even select language and country to access the website?
More posts by @Hamaas447
2 Comments
Sorted by latest first Latest Oldest Best
You need to use a crawler with a cookie jar. Here's one I wrote some time ago that can log in and keep cookies for a site. You didn't mention a language. This is PHO and Mysql or Oracle.
github.com/Pamblam/Crawler
Search Engines Bots such as Googlebot do not use cookies and if your content is only visible with cookies then your content is not crawlable. You need to ensure that the website is crawlable without cookies.
The simple solution is to do a cookie check before serving 'choose language option', if the cookie tests negative then you have your website serve the most popular version of your website, then for other languages you use rel="alternate" hreflang="en-XXX" in the head so Google indexes all versions.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.