This same question was asked on Stack Overflow.

The answer is that the Facebook scraper does not obey robots.txt or support a crawl delay. The best that you can hope for is implementing some sort of server side throttling. Some code for doing that has been provided as one of the answers on Stack Overflow.

10% popularity Vote Up Vote Down

@Moriarity557

Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API.

10% popularity Vote Up Vote Down

Feed

: Facebook and Crawl-delay in robots.txt? Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?

More posts by @Dunderdale272

: Forcing adverts via apache configuration file I have a server I am trying to force adverts for free users. I can do it with a .htaccess file with this line "php_value auto_append_file '/var/www/vhosts/demoblog/demoblog.tk/

: Storing Downloadable Content Securely For a site where users can purchase videos for download. What would be the best way to store the media? Any other helpful considerations are more than welcome.

: Basic tracking of conversion from Adwords using referrer, is it possible? (both Ads in SERP and in Content Network) Reading in AdWords Help it seems the only way to track clicks to my site

: Moving this to an "answer" of sorts for no other reason than the comments aren't long enough: To iterate my comment: Even though DNN supports multiple sites on one install, we never do

Login to post a comment!

2 Comments

Back to top | Use Dark Theme