report_question_error.php?question_id=&url=

Posted in: #RobotsTxt #Sitemap #WebCrawlers #XmlSitemap

I have a robots.txt with the following in it:

User-agent: *
Disallow: tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: tests.compkerworld.com/sitemap.xml

My sitemap.xml, which is auto-generated, has all the URLs even those disallowed using robots.txt.

I have no problem if the search engines crawl all the pages, but I want to know if such an entry will create any kind of conflict for crawlers.

One more question: is the above syntax for disallowing correct?

10.01% popularity Vote Up Vote Down

: Will Google omit my site from mobile search results if I do not have a mobile optimized site? I have heard rumors that if I do not have a mobile-optimized web-site (whether through responsive

@Margaret670

Posted in: #Google #GoogleSearch #Mobile #ResponsiveWebdesign #Seo

3 Comments

: How can I make these Google Analytics numbers add up? (Frequency & Recency) Here's a screen shot from Google Analytics. It's my last months traffic, and this is the 'Frequency & Recency'

@Margaret670

Posted in: #Google #GoogleAnalytics

1 Comments

: Changing title and meta tags via jquery. will it work for crawlers? We have a scenario where we are getting our news articles from database on runtime. So we are chnaging our meta tags and

@Margaret670

Posted in: #GoogleSearch

0 Comments

: Remove all results from google search and restart from scratch the problem with my website is that it has many 404 pages on google , that's why I wanted to remove all the pages from google

@Margaret670

Posted in: #GoogleSearch #GoogleSearchConsole

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Kristi941

Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.

10% popularity Vote Up Vote Down

Feed

: Will there be a conflict between my Sitemap.xml and robots.txt? I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=&url=

More posts by @Margaret670

: Will Google omit my site from mobile search results if I do not have a mobile optimized site? I have heard rumors that if I do not have a mobile-optimized web-site (whether through responsive

: How can I make these Google Analytics numbers add up? (Frequency & Recency) Here's a screen shot from Google Analytics. It's my last months traffic, and this is the 'Frequency & Recency'

: Changing title and meta tags via jquery. will it work for crawlers? We have a scenario where we are getting our news articles from database on runtime. So we are chnaging our meta tags and

: Remove all results from google search and restart from scratch the problem with my website is that it has many 404 pages on google , that's why I wanted to remove all the pages from google

Login to post a comment!

1 Comments

Back to top | Use Dark Theme

: Will there be a conflict between my Sitemap.xml and robots.txt? I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=*

More posts by @Margaret670

: Will Google omit my site from mobile search results if I do not have a mobile optimized site? I have heard rumors that if I do not have a mobile-optimized web-site (whether through responsive

: How can I make these Google Analytics numbers add up? (Frequency & Recency) Here's a screen shot from Google Analytics. It's my last months traffic, and this is the 'Frequency & Recency'

: Changing title and meta tags via jquery. will it work for crawlers? We have a scenario where we are getting our news articles from database on runtime. So we are chnaging our meta tags and

: Remove all results from google search and restart from scratch the problem with my website is that it has many 404 pages on google , that's why I wanted to remove all the pages from google

Login to post a comment!

1 Comments

Back to top | Use Dark Theme

: Will there be a conflict between my Sitemap.xml and robots.txt? I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=&url=