How to Block spiders,crawlers, and bad search agents

Apr 3, 2011 at 2:56 PM
Edited Oct 24, 2011 at 6:46 PM

Anyone have an extension or custom code that does this for blogengine?

I would like to make a list of bots to block from website.

Java Blog

Apr 3, 2011 at 6:37 PM


the first thing you can do ist to add restrictions to you robots.txt file in your be root folder. The "good" bots will accept your restrictions. If this does not solve your problem you can just block certain ip addresses within your webserver or you have to write a bit of code that could avoid crawling your website. Maybe this would be a good extension for be in form of a http handler. I don't know if such a solution already exists for

Apr 4, 2011 at 3:56 AM

Currently working on putting


into an extension.