: Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string? I know there are a lot of similar questions and I searched around but I never

@Sherry384

Yes. Search engines and any crawler that is coded properly can crawl pages with parameters and index them as separate pages. The fact that the crawler you referenced/used cannot handle this is an indication that I would not use this tool again.

I often recommend using www.screamingfrog.co.uk/seo-spider/ if you can access your site directly. This tool allows 500 pages free and is rather affordable if you wish to pay for it. ScreamingFrog is very good at reporting a long list of issues that any webmaster might be concerned with including most if not all of what the link you provided supplies.

If you do use this tool or any tool like this, I suggest turning of AdSense ads or any other ads you are using so that you do not get into trouble.

Vote Up Vote Down

@Heady270

Googlebot and other search engine robots will crawl URLs with a variety of query strings by default. All they need is a link to each, or a sitemap that contains each. Google considers URLs different even when they differ only by the query string.

Google Webmaster Tools allows you to customize this behavior for your site. If you have a query parameter that doesn't actually change the page, you can tell Google about it in the site settings. Then Google may crawl only one representative URL rather than each value of the parameter.

Vote Up Vote Down