: Need to Stop Bots from Killing my Webserver I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of

@Yeniel560

I've solved the same issue with the script available at perishablepress.com/blackhole-bad-bots/. With this blackhole approach I collected a list of malicious ip, and then using .htaccess denied them. (Which is not mandatory, since the script itself does the banning. But I need to reduce the server load by avoiding php parsing for known unwanted ips) in three days my traffic came down from 5GB per day to 300MB, which is quiet expected.

Check this page also for full list of htaccess rules to block many known junk bots. www.askapache.com/htaccess/blocking-bad-bots-and-scrapers-with-htaccess.html

Vote Up Vote Down

@Sue5673885

Go set up your domain on Cloudflare (free service for this). They block malicious bots at the domain level before they hit your server. Takes about 20 minutes, never have to monkey with the code.

I use this service on all my sites and all client sites. They identify malicious bots based on a number of techniques including leveraging project Honey pot.

Vote Up Vote Down

@Margaret670

PHP Limit/Block Website requests for Spiders/Bots/Clients etc.

Here i have written a PHP function which can Block unwanted Requests to reduce your Website-Traffic. God for Spiders, Bots and annoying Clients.

CLIENT/Bots Blocker

DEMO: szczepan.info/9-webdesign/php/1-php-limit-block-website-requests-for-spiders-bots-clients-etc.html
CODE:

/* Function which can Block unwanted Requests
* @return boolean/array status
*/
function requestBlocker()
{
/*
Version 1.0 11 Jan 2013
Author: Szczepan K
www.szczepan.info me[@] szczepan [dot] info
###Description###
A PHP function which can Block unwanted Requests to reduce your Website-Traffic.
God for Spiders, Bots and annoying Clients.

*/

$dir = 'requestBlocker/'; ## Create & set directory writeable!!!!

$rules = array(
#You can add multiple Rules in a array like this one here
#Notice that large "sec definitions" (like 60*60*60) will blow up your client File
array(
//if >5 requests in 5 Seconds then Block client 15 Seconds
'requests' => 5, //5 requests
'sek' => 5, //5 requests in 5 Seconds
'blockTime' => 15 // Block client 15 Seconds
),
array(
//if >10 requests in 30 Seconds then Block client 20 Seconds
'requests' => 10, //10 requests
'sek' => 30, //10 requests in 30 Seconds
'blockTime' => 20 // Block client 20 Seconds
),
array(
//if >200 requests in 1 Hour then Block client 10 Minutes
'requests' => 200, //200 requests
'sek' => 60 * 60, //200 requests in 1 Hour
'blockTime' => 60 * 10 // Block client 10 Minutes
)
);
$time = time();
$blockIt = array();
$user = array();

#Set Unique Name for each Client-File
$user[] = isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : 'IP_unknown';
$user[] = isset($_SERVER['HTTP_USER_AGENT']) ? $_SERVER['HTTP_USER_AGENT'] : '';
$user[] = strtolower(gethostbyaddr($user[0]));

# Notice that i use files because bots does not accept Sessions
$botFile = $dir . substr($user[0], 0, 8) . '_' . substr(md5(join('', $user)), 0, 5) . '.txt';


if (file_exists($botFile)) {
$file = file_get_contents($botFile);
$client = unserialize($file);

} else {
$client = array();
$client['time'][$time] = 0;
}

# Set/Unset Blocktime for blocked Clients
if (isset($client['block'])) {
foreach ($client['block'] as $ruleNr => $timestampPast) {
$left = $time - $timestampPast;
if (($left) > $rules[$ruleNr]['blockTime']) {
unset($client['block'][$ruleNr]);
continue;
}
$blockIt[] = 'Block active for Rule: ' . $ruleNr . ' - unlock in ' . ($left - $rules[$ruleNr]['blockTime']) . ' Sec.';
}
if (!empty($blockIt)) {
return $blockIt;
}
}

# log/count each access
if (!isset($client['time'][$time])) {
$client['time'][$time] = 1;
} else {
$client['time'][$time]++;

}

#check the Rules for Client
$min = array(
0
);
foreach ($rules as $ruleNr => $v) {
$i = 0;
$tr = false;
$sum[$ruleNr] = '';
$requests = $v['requests'];
$sek = $v['sek'];
foreach ($client['time'] as $timestampPast => $count) {
if (($time - $timestampPast) < $sek) {
$sum[$ruleNr] += $count;
if ($tr == false) {
#register non-use Timestamps for File
$min[] = $i;
unset($min[0]);
$tr = true;
}
}
$i++;
}

if ($sum[$ruleNr] > $requests) {
$blockIt[] = 'Limit : ' . $ruleNr . '=' . $requests . ' requests in ' . $sek . ' seconds!';
$client['block'][$ruleNr] = $time;
}
}
$min = min($min) - 1;
#drop non-use Timestamps in File
foreach ($client['time'] as $k => $v) {
if (!($min <= $i)) {
unset($client['time'][$k]);
}
}
$file = file_put_contents($botFile, serialize($client));


return $blockIt;

}


if ($t = requestBlocker()) {
echo 'dont pass here!';
print_R($t);
} else {
echo "go on!";
}

Vote Up Vote Down

@Gretchen104

To continue on smassey's post, you can put several conditions:

RewriteCond %{HTTP_USER_AGENT} ^GoogleBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^OtherAbusiveBot
RewriteRule ^rss2html.php$ /static.html [L]


This way, the bots still access your pages, but just not that one. As it's strange that the (legitimate) bots are not keeping to rules, do you have any referers that push bots to your page from other sources (domainname forwarding, ...)

Vote Up Vote Down

@Ogunnowo487

// Checks to see if this script was called by the main site pages,
// (i.e. index.php or mobile.php) and if not, then sends to main page
session_start();
if (isset($_SESSION['views'])) {$_SESSION['views'] = $_SESSION['views']+ 1;} else {$_SESSION['views'] = 1;}
if ($_SESSION['views'] > 1) {header("Location: website.com/index.php );}


This script does not do what the comment says, in fact it does the complete opposite. This will always let the bots through, since the session variable will never be set when the bot requests your script. All it will potentially do is prevent legitimate requests (from index.php or mobile.php) from calling the script more than once.

In order to prevent a bot from accessing your script you should only allow access if a session variable (or cookie) is actually set. Assuming of course that the (malicious) bot does not accept cookies. (We know that the real Googlebot doesn't.)

As has already been mentioned, placing rss2html.php above the web root (outside of the public webspace) would prevent a bot from accessing the script directly - but you say this causes other problems? Or, place it in a directory and protect that directory with .htaccess. Or you might even be able to protect just the file itself in .htaccess from direct requests?

Vote Up Vote Down

@Cugini213

If rss2html.php is not being used directly by the client (that is, if it's PHP always using it rather than it being a link or something), then forget trying to block bots. All you really have to do is define a constant or something in the main page, then include the other script. In the other script, check whether the constant is defined, and spit out a 403 error or a blank page or whatever if it's not defined.

Now, in order for this to work, you'll have to use include rather than file_get_contents, as the latter will either just read in the file (if you're using a local path), or run in a whole other process (if you're using a URL). But it's the method that stuff like Joomla! uses to prevent a script from being included directly. And use a file path rather than a URL, so that the PHP code isn't already parsed before you try to run it.

Even better would be to move rss2html.php out from under the document root, but some hosts make that difficult to do. Whether that's an option depends on your server/host's setup.

Vote Up Vote Down

@Ann8826881

You should really make sure in a first place that any page requested from a useragent, of whichever abusive crawler you have, will be served a static page.

An apache mod_rewrite with a condition or equiv with your http server. For apache, something like this:

RewriteCond %{HTTP_USER_AGENT} ^GoogleBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^OtherAbusiveBot
RewriteRule ^/$ /static_page_for_bots.html [L]

Vote Up Vote Down

@Hamaas447

It's likely that your site is being indexed by fake google bot(s). You can try adding a check and serve 404 for all fake google bot requests.

Here is an article that explains how to verify Googlebot: googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html
Also you could check your records against known fake bots: stopmalvertising.com/security/fake-google-bots.html

Vote Up Vote Down

@Martha676

You could set your script to throw a 404 error based on the user agent string provided by the bots - they'll quickly get the hint and leave you alone.

if(isset($_SERVER['HTTP_USER_AGENT'])){
$agent = $_SERVER['HTTP_USER_AGENT'];
}

if(preg_match('/^Googlebot/i',$agent)){
http_response_code(301);
header("HTTP/1.1 301 Moved Permanently");
header("Location: www.google.com/ );
exit;
}


Pick through your logs and reject Bingbot, etc. in a similar manner - it won't stop the requests, but might save some bandwidth - give googlebot a taste of it's own medicine - Mwhahahahaha!

Updated

Looking at your code, I think your problem is here:

if (preg_match("/$bot[1]/i",$hostname) && $ipaddress == $iphostname)


If they are malicious bots then they could be coming from anywhere, take that $ipaddress clause out and throw a 301 or 404 response at them.

Thinking right up by the side of the box


Googlebot never accepts cookies, so it can't store them. In fact, if you require cookies for all users, that's probably going to keep the bot from accessing your page.
Googlebot doesn't understand forms - or - javascript, so you could dynamically generate your links or have the users click a button to reach your code (with a suitable token attached).

<a href="#" onclick="document.location='rss2html.php?validated=29e0-27fa12-fca4-cae3';">Rss2html.php</a>


rss2html.php?validated=29e0-27fa12-fca4-cae3 - human
rss2html.php - bot

Vote Up Vote Down