: How to allow google to index my deep pages? Let's say I'm building an open wiki system. Each user can start a new page, give it a name like "how-to-travel-faster-than-light" which will turn

@Shakeerah822

How would google ever find and index the "how-to-travel-faster-than-light" page if it can't find it, ...


If it can't find it; it can't index it. It's that simple. However, Google can potentially discover URLs in many ways, from scanning emails in Gmail to URLs typed into its Chrome web browser (although these methods are naturally unreliable if you are trying to get a URL indexed).


How did wikipedia or stack exchange solve this problem?


Well, Wikipedia does have numerous Contents and hierarchy Indices, as well as a complete alphabetical index - so it would certainly seem spiderable. It is also very well cross-linked and inbound links are second to none. It might even have a sitemap (although I can't find it), as it's still well within the 2.5 billion sitemap URL limit (2017 figures) as set by Google.

The Stack Exchange Questions page - that lists new questions first (and active questions rise to the top) is naturally a spiderable index of all pages/questions on the site. There is also an RSS question feed. And, until recently, there was an XML sitemap. (The sitemap URL stated in the Webmasters robots.txt seems to result in a 404 currently?)


There is no site map listing every link to every user page.


Why not? This is a standard way of informing search engines about hard to reach pages.

When pages are created you can ping Google and other search engines to notify them of an update. They will then request your updated (auto-generated) sitemap.

Just having Google Analytics installed on your site will at least allow Google to discover URLs the very first time you (or the author) visits the page.

Vote Up Vote Down