: With DNS A records, you cannot set records for a specific file. DNS is all or none for the domain. If you must serve a robots.txt file, then you have to have DNS resolve to a valid

With DNS A records, you cannot set records for a specific file. DNS is all or none for the domain.

If you must serve a robots.txt file, then you have to have DNS resolve to a valid domain and have a web server that will answer that request.

Here is what I recommend:

DNS

Setup a CNAME to point domain.com to domain.com.
Setup an A record to point domain.com to your server's IP address.

Web Server

On your server (assuming Apache), you can use .htaccess to limit what files are being served.

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI} !/robots.txt [NC]
RewriteRule ^(.*) - [L,R=404]

This will return a 404 (or any code you want) for all pages but /robots.txt.

This way you can host robots.txt but not any other pages.

There is also the 410 Gone HTTP response code but I am not sure how bots handle it.

10.4.11 410 Gone

The requested resource is no longer available at the server and no
forwarding address is known. This condition is expected to be
considered permanent. Clients with link editing capabilities SHOULD
delete references to the Request-URI after user approval. If the
server does not know, or has no facility to determine, whether or not
the condition is permanent, the status code 404 (Not Found) SHOULD be
used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web
maintenance by notifying the recipient that the resource is
intentionally unavailable and that the server owners desire that
remote links to that resource be removed. Such an event is common for
limited-time, promotional services and for resources belonging to
individuals no longer working at the server's site. It is not
necessary to mark all permanently unavailable resources as "gone" or
to keep the mark for any length of time -- that is left to the
discretion of the server owner.