: How did Google find my unlinked newly created pages? I am building a new site, but creating the content using a subdirectory on an existing site. I have never posted a link to this subdirectory

@Si4351233

Please post here the Robots.txt codes you used to block the access.
Google cannot crawl website unless you have updated your robots.txt later after launching those pages because Google does crawl quickly and it has index the URL before you have actually updated your robots file. If it has happened then wait, the Google will automatically de-index your URL.

Vote Up Vote Down

@Megan663

I think when using Chrome or Google Toolbar, Googlebot will follow you while browsing your own website. So it will reach pages that you reached, regardless of links pointing to them.

About removing the content from Google index, I think that takes a while.

Vote Up Vote Down

@Heady270

Many CMS systems such as WordPress tell Google about each page that is published. WordPress calls this "update services" and uses ping-o-matic by default.

Google may also use data it gets from browsers to start crawling. If you are using a browser with a Google Toolbar (or PageRank checker), then Google gets a list of all the pages that you visit. However, Google denies that they use toolbar data for this purpose.

Google does say that a common way for "secret" URLs to be discovered is for them to link out to other sites. Those other sites then see the "secret" page in the referrer and sometimes publish a list of referrer links (a common feature of blogs).

Vote Up Vote Down