: How to remove a fragment / hash URL from Google search results? My old site used to write the URLs without !. Eg: www.example.com/#/products But it has been more than 6 months since I changed

@Jessie594

First, you don't have to go through the trouble of removing the old URLs with the hash. If your content is getting crawled and indexed appropriately under the new hash-bang URL, it'll soon get replaced.

It's not entirely clear what the problem is without troubleshooting against the live site, but hopefully these steps will help you discover a solution.

You'll want to start by eliminating common problems.

Check if Googlebot is Accessing Your HTML Snapshots

The whole idea of AJAX crawlability is to get your product images, descriptions, and specifications indexed. I'm assuming you've followed the appropriate instructions from Google's AJAX crawlability specifications, so you'll want to make sure that Googlebot is able to see your HTML snapshots. If Google isn't seeing the correct HTML snapshots, this is likely the reason why Google hasn't updated it's index to reflect the new hashbang URLs you created.

Using Google Webmaster Tools (GWT), go to the "Crawl » Fetch as Google" section. Then have Goolgebot fetch and render the full _escaped_fragment_ URL => example.com/?_escaped_fragment_=products. If all is well, you should see the status as "Complete". You should also be able to click on the status, to see the HTML snapshot of the particular product.

If you see a "Not Found" for the status or "Blocked", that's likely the reason Google hasn't updated the URLs in the index.

If it's "Blocked", that means you have a rule in your robots.txt file that is preventing Googlebot from accessing the HTML snapshot.

If it's "Not Found", that means you probably haven't configured nginx to appropriate route requests with ?_escaped_fragment_= to the server side rendering of the HTML snapshot.

If you see a "Partial", then you'll need to click through on the status, and see the various resources which are blocked from Googlebot. But basically, if you don't see the appropriate content in the thumbnail image from the Google's render of the page, it's likely that you're blocking some resource that Google needs to render out the content.

Check if Angular is Rendering the HashBang URLs in the HREFs

If the rendered thumbnail of the page is correct, we've established that Googlebot is able to see the server side rendered HTML and the routing is correct for the _escaped_fragment_ requests. This then points to the front end since Googlebot doesn't seem to be discovering the new hashbang URLs.

A quick check of your access logs should help you confirm this issue. If you have Splunk, LogStash, or good ole grep handy, you'll want to look for requests that have the following criteria:


user agent contains "googlebot"
request URL contains "?_escaped_fragment_=products"
requesting IP starts with "66.249"


Basically, you want to look for all verified Googlebot requests. Googlebot mostly comes from 66.249.*.* these days.

If you're not seeing many requests compared to the # of Pages Crawled Per Day listed under "Crawl » Crawl Stats" in GWT, then it's likely that Googlebot isn't able to see the hashbang URLs in the href attributes of the anchor tags.

If you navigate to example.com/#!/products and Inspect Element in Chrome or Firefox, you'll want to make sure that the rendered HTML has clear anchor tags with hrefs that include the hashbang versions of the URLs. If not, then you'll want to reconfigure Angular to do so.

If you do see them there, you'll want to then want to fetch and render example.com/#!/products or another page like your homepage that would have anchor tags with href attributes linking to hashbang URLs.

When you click on the status to see the "rendered" version of the page, you can click on the "Fetching" tab which is next to the default "Rendering" tab, to see the HTTP response headers and resulting HTML.

If you don't find the hashbang references in the appropriate spots from the "Fetching" tab, you'll have to step through your Angular stack to see where the disconnect is happening. Likely you're doing a javascript redirect in an asset file that is blocked by Googlebot, and it's probably not seeing the updated references.

When All Else Fails

If it's none of the things above, well then it's more likely a bug with Google itself. This actually happens from time to time, and although Googlers are smart folks, they do get it wrong sometimes.

You'll want to jump into the Google Webmaster Forums: Crawling, index, and ranking, and hopefully one of the many Googlers who frequent the forums will investigate it for you.

An Alternate Solution

So after all that's been said and done, I strongly recommend against using the _escaped_fragment_ standard. Although it's something that Google put out there and it's been adopted by Bing, you rarely see sites implement it correctly, and you rarely see the hashbang URLs appear in the top search results for indexed pages.

My suggestion, if you have the engineering time & resources to do it, is to move to using pushState(). With pushState you'll need to add more logic to your routes to handle both client and server side requests, but you can then take advantage of using canonical tags and you don't have to mess with the ?_escaped_fragment_ overhead.

I know Angular + pushState isn't the most well documented thing out there, but there are some good posts out there depending on the rest of your stack.

Lastly, I don't want to be overly self promotional, but as these things go, the conversation eventually leads to "pre-render" vs "server side" rendering. I did a talk at SMX Advanced covering some common AJAX crawlability issues. If you jump to slide 34, you can see some of the pros and cons. Also on slide 49 and 50 there are references to a few other good Angular + SEO posts.

Vote Up Vote Down

@Cugini213

If you have changed your URLs you need to 301 redirect OLD urls to new one, Once you do this Google will automatically update its indexing status within a month or so depending on the crawling rate of your website.

Here is a complete guide to 301 redirection
moz.com/learn/seo/redirection

Vote Up Vote Down

@Berryessa370

It's been a while since i have been in google webmaster's tools but I do remember that in there is a tool for removing indexed pages for any website..

**You will need to link the website with the particular webmaster's tool account to do so...
www.google.com/webmasters/tools/dashboard
login or create new account (link the website as mentioned above)

click on google index & select remove urls

on the opening page insert all urls you want removed.....**

Vote Up Vote Down