: How do I prevent bots from creating links to pages that do not exist? I just installed Sitemap Creator and run a test. I have a newly developed site using PhalconPHP framework. The issue that

@Eichhorn148

Googlebot can create non-existing URLs and try to crawl them too, so you should strongly avoid that non-existing URLs send on requesting any server answer beside of 404.

Define your 404 document and force 404 server answer for any request of non-existing URLs. I.e. with following htaccess rules:

ErrorDocument 404 /404.php
AcceptPathInfo Off

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /404.php [NC,L]
</IfModule>

Vote Up Vote Down

@Pope3001725

Fix your website. Bots don't create links. They follow them. If they can find links to pages that don't exist then your website has links to pages that don't exist*. Somewhere, or in multiple places, you have code that is creating a hyperlink that is not valid. You should check your logs to see what the referring page is for those invalid URL requests as that will point you in the right direction.

*Links from third party sites can link to non-existent pages on your site, too, bit that is out of the scope of this question.

Vote Up Vote Down