: Good tool to crawl my site and help me find dead link and unlinked files I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database,

@Vandalay111

Link Examiner is a really good freeware too for your need.

Vote Up Vote Down

@Nimeshi995

There are several products from Microsys, especially their A1 Sitemap Generator and A1 Website Analyzer that will crawl your website and report everything you can possibly imagine about it.

That includes broken links, but also a table view of all your pages so you can compare things like identical <title> and meta description tags, nofollow links, meta noindex on webpages, and a whole lot of diseases that just need a sharp eye and a quick hand to fix.

Vote Up Vote Down

@Cugini213

Try W3C's open source tool Link Checker. You can use it online or install it locally

Vote Up Vote Down

@Mendez628

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free.

The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.

Vote Up Vote Down

@Dunderdale272

I'm a big fan of linklint for linkchecking large static sites, if you have a unix command line around (I've used on linux, MacOS, and FreeBSD). See their site for installation instructions. Once installed, I create a file called check.ll and do:

linklint @check .ll


Here's what my check.ll file looks like

# linklint
-doc .
-delay 0
-http
-htmlonly
-limit 4000
-net
-host example.com -timeout 10


That does a crawl of example.com and generates HTML files with cross-referenced reports for what is broken, missing, etc.

Vote Up Vote Down

@Murray432

I've used Xenu's Link Sleuth. It works pretty well, just be sure not to DOS yourself!

Vote Up Vote Down