: Is Google indexing pages that has no connection with other pages? I am working on a web project that has nearly 100 thousand instant users and there is a webpage that we are using for test

@YK1175434

Googlebot (the web crawler of Google) can discover your page by different ways:


if it finds a link to your page from a web page (inside or outside your domain name)
if you submit the URL to your Google Webmaster Tools account through a sitemap.xml


In that case, the page can be indexed. In most of cases, Googlebot decides to index it (for example, it's not the case when your page has a lot of duplicate content with another page).

If you don't want this page indexed, just put this <meta name="robots" content="noindex, nofollow"> tag in your page and block the page with your robots.txt like this:

User-agent: Googlebot (or * for all web crawlers)
Disallow: /the-page

Vote Up Vote Down

@Gretchen104

If web crawlers get to this page one way or another, and if it is not marked as noindex and it is not blocked in your robots.txt, then yes, they may decide to index it.

You may also use noarchive to make sure they don't keep a cached version of it.

Vote Up Vote Down

@Rivera981

No, Google doesn't automatically index your pages, if no other site links to them. You should put a robots.txt file on the root directory of your server, which can prevent the search engine from indexing the pages of a specific folder.

It could look like this:

User-agent: *
Disallow: /

Vote Up Vote Down