: Generating a list of URLs including metadata Basically I want to do an editorial sweep on my website, I need a program to crawl my website and generate a list of URLS and their metadata.

@BetL925

You can also try Site Visualizer tool. It has no limitations on number of URLs to crawl in trial version - it's fully-functional in 30 days.

After downloading and installing (Standard version would be enough for your task), click Project -> New, specify a URL of the website you need to crawl, then click Start Crawl tool button.

When the crawling is finished, right-click on the Pages table (Pages & Links tab), then click Export context menu command:


In the save dialog appears, choose the destination of the CSV file, specify delimiter (comma by default, or use semicolon to adjust columns for MS Excel's sheet). URLs of all pages of the website will be exported into specified CSV file.

To export certain columns of the Pages table, use the Visible Columns dialog of the mentioned above context-menu before the export:

Vote Up Vote Down

@Ravi8258870

I would highly recommend the Screaming Frog SEO Spider. He gets you all URLs, Status Codes, Titles, Descriptions, Images on your Website, etc.
It's really a great SEO tool, as far as I know most SEOs use it.
You can export csv- and Excel-files, as you requested, and also you can create sitemaps, image-sitemaps, and so on.
The problem here is that the free version you it's limited to crawl 500URLs. It's not that expensive to buy the full version, most people do so.

Another thing you can try is the URL and Meta-Tag Extractor or the Yenu Link Sleuth Tool, but I'm not sure if they can fulfill your needs. I don't have experience with these, but they can show you URLs and their meta-tags, but I don't know if either of them can export to csv-files.

Greets.

Vote Up Vote Down