: What are some client-side sitemap generators? Most of the sitemap generators I've found all scan my internal files and base it on that. However using apache and htaccess I have many aliases

@Gail5422790

GSiteCrawler is my go to tool for this. You can schedule crawls and have it automatically upload the new files.

And it can generate a host of other useful reports.

Vote Up Vote Down

@Kimberly868

You could consider trying A1 Sitemap Generator. Just make sure to uncheck "simplified easy mode". Then you can control the crawler with the output and analysis filters handling URLs matching with strings, paths, regular expressions, file extensions and mime types.

You can automate sitemap generator using scheduler / command line.

Vote Up Vote Down

@Shanna517

The best solution is to just generate dynamic sitemaps server-side. You have access to all of the application code and rewrite rules, so it doesn't really make sense to use a spidering tool to crawl your site. zenphoto has a plugin for generating sitemaps, as do most other CMSes. Just use a sitemap index to link to those sitemaps.

But if you still want to use a webcrawler, there are a few listed here.

Some people at Google have also developed a pretty ingenious sitemap tool called the Google Sitemap Generator. Instead of spidering the site (which is just the same thing that Google/Bing/Yahoo already do), it monitors server traffic to see how users access your pages and creates sitemaps that way.

Vote Up Vote Down