: Google references subfolders Last week I launched a website and created a robots.txt: User-agent: * Disallow: Sitemap: http://mywebsite.tld/sitemap_robots.xml Now I've checked the indexed pages (Google

@Karen161

OK, I've finally found a plausible explanation, why Google has indexed folders and files outside of the document root:



The website I've launched last week is a second version of a project. The first version was Joomla! based and its dosument root was to the project root:


/var/www/.../mywebsite.tld


The second version bases on Zend Framework 2 and uses its common project structure. So the web root is


/var/www/.../mywebsite.tld/public


We had some technical troubles. So the initial deploy of the second version took several days and only at the last day the document root of the website has been changed to .../public. During all this time the index page was looking like this:



So the Google bot indexed all these links, that are broken now, but was returning status 200 some days ago.

Vote Up Vote Down

@YK1175434

In order to block all robots in your website, the correct code is:

User-agent: *
Disallow: /


Don't forget the / (slash) after Disallow:.

To block only subdirectories and their internal webpages with robots.txt, you have to list them one by one. Thus you need to know their names:

User-agent: *
Disallow: /vendor/
Disallow: /module/
...


To understand how robots.txt works, you can read this or this.



Moreover, for your information, when you buy a domain name, several websites are notified and put links to your website even if you don't want to. That's why, you need to block all robots if you don't want to see your webpages being indexed.

Vote Up Vote Down