: Determining which sitemap entries not indexed by google We submit a sitemap to google and can see this being indexed in webmaster tools. We have 5140 entries, and have broken this up to 10

@Sue5673885

I will reply as an answer because I cant fit everything in a comment:

You could use a few tools, but I will describe the way I am most comfortable with, PHP + cURL + DOMDocument.

First you need to generate the search queries, so probably reading your sitemap.

$url = 'www.google.com/search?q=' . urlencode("site:".$sitemap_url);


(add more parameters to search url if needed)

Then you probably want to fake your useragent and set other cURL options if needed.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Your fake useragent string here');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$out = curl_exec($ch);

$dom = new DOMDocument(); @ $dom->loadHTML($output);


Now that you have Google search loaded in a DOMDocument, you can parse it and look if your url is actually present in the search. If it is, it's indexed.

Hope this helps

Vote Up Vote Down