Mobile app version of vmapp.org
Login or Join
Lee4591628

: Tool to generate list of redirections (sitemap for .htaccess) Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version

@Lee4591628

Posted in: #301Redirect #Ubuntu #UrlRewriting #WebCrawlers

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site?

I'm building new version of some site, but I don't want to get 404 errors for the old pages. Instead I want to return 301 redirection, but first of all I need the list of all the old links, then redirect to new ones.

The only one I've found so far is xenu webcrawler, but it's for Windows.

10.02% popularity Vote Up Vote Down


Login to follow query

More posts by @Lee4591628

2 Comments

Sorted by latest first Latest Oldest Best

 

@Sue5673885

You don't have to use software, you can use .htaccess redirects that follow patterns like if the old link is to /software-for-learning/softwareitem-21, you could have it read the directory structure and forward everything inside of that (in this case /softwareitem-21 is ignored, since you matched the pattern based on the top level directory) redirect to the new path pattern (eg should now go to /products/software/learning/.

This is a bit tougher, I won't lie, but it might save you a few headaches and knowing how to write these rules out to follow certain patterns (like finding the numbers inside the title, or ignoring the title and just redirecting the page as long as its inside the directory /A/ to /new/directory/structure/a/ or whatever).

more info here: www.javascriptkit.com/howto/htaccess7.shtml

10% popularity Vote Up Vote Down


 

@Murray432

Highly recommend Xenu if you can run a virtual windows system - it's very quick and very simple.

That said this would appear to be an extensive list, I'm told harvest is the best of the bunch, but can be complicated to set up.

You could also use platform independant software like www.xml-sitemaps.com/.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme