: Adding URL parameters to Google Webmaster Tools: URL encoded or not? When adding parameters to Webmaster Tools for URLs that I don't want indexed, should I add the parameters with actual brackets,

@Eichhorn148

Googlebot expects URL parameter names to be escaped when adding them via Google Webmaster Tools.

I ran an experiment on my site where I created a page with two different URL parameters: unesc% and escap%. I created page for the experiment on my site. On it uses the parameters with four values each

I then added escap%25 and unesc% to Google Webmaster Tools as "Effect: None, Crawl: Representative URL" and sat back to watch what would happen.

The first thing I noticed was that Googlebot actually came to my site and crawled all the parameters, despite the webmaster tools setting. This is either because:


There is a bug in Googlebot where it doesn't respect the settings
They designed Gogolebot not to respect the settings
It takes some time for Googlebot to become aware of the settings. (I put the settings in before Googlebot had a chance to crawl, but it started crawling within an hour).


After Googlebot crawled, webmaster tools didn't report that Googlebot was monitoring any matching URLs. I figured that it would take some time before GWT started showing what Googlebot was actually doing so I waited. A couple weeks later I now get this:



I would still suggest adding both to your site just as a precaution (I'm only linking to the escaped version of the URL, I'm not user if it would be different if I linked to the un-escaped version).



Here is the testing that I had done earlier:

I tested it by adding both, just to see how the tool would react.



I thought that it might give me an error for one of the two, or possibly canonicalize them to the same thing. I could actually add both of them (as you can see from the screenshot).

That leads me to guess that the un-escaped one is the one that would match while Googlebot is crawling. But, that is just a guess. Since it allows you to add both, you might as well just go ahead and add both to be safe.

Vote Up Vote Down