: Is there a way to download the Google index for my website? I have a client that's launching a new website on their current domain. I don't want to lose their SEO rankings, so I need to

@Sherry384

There are several spiders that will create an XML sitemap that you can use to then audit your work. I used one once years ago, but I cannot find which one I used so I do not feel comfortable making a recommendation. However, you can search: www.google.com/?gws_rd=ssl#q=xml+sitemap+generator+software+free I would avoid the online options and go for a software install. You will notice I used the word free in the search query. You can delete this of course. You would have to have access to the web site and can spider the site fully for this to help.

If you do not see a free option that you like, then I can highly recommend: www.screamingfrog.co.uk/seo-spider/ This is a yearly license that will help audit your site and create an XML sitemap. It will also help with SEO of the new site. I use other tools, but Screaming Frog does have an excellent reputation and was impressed when I tested it. It would be the next tool I would add to my personal arsenal.

I suspect that Screaming Frog will have an option for you and worth taking a look at.

Vote Up Vote Down

@Annie201

When involved from the start

When migrating I keep the existing URLs in mind right from the start of the project. Like that I already have most links covered when switching.

Right after the switch I check site:example.com and see if most links there are still working. If a lot are failing I go through them manually if there are less than say 300.

In the weeks and months after the switch I keep an eye on Google Webmaster to find broken URLs.

When coming in later

If I would not be involved in the migration from the start and I'd have doubts I'd probably write some code in Python (e.g. with Selenium) or CoffeeScript to either parse the result of a Google query on site:example.com or work with the logs from the webserver. If you have logs for a couple of months that will probably cover 99% of the URLs indexed by Google.

Whatever is indexed by Google will be checked by Google regularly. If you have logs for the past years you can simply grep on Googlebot to get a list of all URLs accessed by Google. This can get you started: zcat access*.log*|awk '{ print }'|sort|uniq (probably needs some sed and a different $n for awk).

Of course I'd first check if such code is already available under a free license.

Naturally I'd still check Google Webmaster and run the site:example.com query manually as described above.

Vote Up Vote Down

@Nimeshi995

Method One - Google search

You can find out what Google has indexed by returning results the exact results by searching for site:http://www.example.com. You should be aware that this method may not track very new URLS or if you have a lot of pages then Google will just return a few hundred.

Method Two - sitemap

There might be a sitemap.xml already on the server and you should be able to use that but you should also be aware that if the sitemap is not updated it may be obsolete. You should do both sitemap.xml and site:http://www.example.com just to confirm it is.

Method three - low volume of URLS (pen and paper)

Another favorable method is simply navigate through the website and of course write down the URLS. After your done you should proceed to ensure that you have them all by doing a good old site:http://www.example.com.

Method four - copying old site to a hidden sub domain

Often I will copy old site information to a secured sub domain that requires a password, this is a good way of having a backup and referring to the old site even after the project has been delivered. You can copy the site using FTP and backup the DB using phpmyadmin and then port over to a sub domain. Or alternatively you can convert the site into a static HTML site by using HTTrack.

Updating content is not a guarantee increase in rankings and cab be negative

It's important to note that when changing content on the majority of the website in question that you should inform your client to expect a shift either positive or negative, if the content is high quality, unique and not over optimized you should expect the negative to be temporary. Normally negatives result as too much optimization, over thinking URLS, Titles, and so on results in this way.

Vote Up Vote Down