: If by bots, you mean crawlers, then a simple robots.txt file should be able to correct the problem for bots that actually adhere to it. There is little that can be done to prevent those

@Sue5673885

If by bots, you mean crawlers, then a simple robots.txt file should be able to correct the problem for bots that actually adhere to it. There is little that can be done to prevent those that do not, however, except by blocking them at the network level.

The robots.txt file can be used to whitelist or blacklist crawlers or specific parts of the web site. If you have some static material (or static versions of database-driven copy) that can be served instead, you can point crawlers there without sacrificing too much from a freshness perspective.

Vote Up Vote Down

Login to post a comment!