Differences

This shows you the differences between two versions of the page.

--- networking:linux:bad_bot_trap [2011/11/09 14:51]
gcooper
+++ networking:linux:bad_bot_trap [2013/10/13 14:59] (current)
gcooper
@@ Line 1: / Line 1: @@
 ====== Bad Bot Trap ======
+See also **[[networking:linux:fail2ban|Fail2Ban]]**
 http://www.kloth.net/internet/bottrap.php
-Used here with **[[networking:linux:fail2ban|fail2ban]]**:
+We use Fail2Ban to block bad bots.
+===== robots.txt =====
+Misbehaving bots may access areas of your web site even if you tell then not to.  That's what we key on here.
+:!: First, change to your web root folder.  Possibly ''public_html''.
+Edit robots.txt and add a ''Disallow'' line under ''User-agent: *'':
 <file>
-vim robots.txt
+cd public_html
+vi robots.txt
 User-agent: *
@@ Line 12: / Line 24: @@
 </file>
-<file>
+===== Web Site Header =====
-mkdir /var/www/web1/web/bot-trap
-vim /var/www/web1/web/templates/ja_purity/index.php
+We use a tiny image for embedding the hidden link.  ''blank.png'' should be just a single pixel transparent image and is unimportant (and may already exist).  If it doesn't already exist, you can get ''blank.png'' like this:
+<file>
+cd images
+wget http://www.sonoracomm.com/images/blank.png
+cd ..
 </file>
-Add this in the header.  blank.png should be just a single pixel transparent image and is unimportant.  The link is important.
+Edit your HTML header (''<head>'' section) (''index.html'', ''templates/yourtemplatename/index.php'', etc.) and add this line (modify as necessary):
 <file>
+vi index.html
 <a href="/bot-trap/"><img src="images/blank.png" border="0" alt=" " width="1" height="1"></a>
 </file>
-Now we create a file so as not to pollute the error logs:
+===== bot-trap Folder =====
-<file>
+Now we create the ''bot-trap'' folder and a file so as not to pollute the error logs.  Change the URL to your own domain and the permissions as necessary:
-vim /var/www/web1/web/bot-trap/index.html
+<file>
+mkdir bot-trap
+cat << EOF >> bot-trap/index.html
 <html>
   <head><title> </title></head>
@@ Line 36: / Line 56: @@
   </body>
 </html>
+EOF
+chown -R apache.apache bot-trap
 </file>
-Add another regex to the fail2ban filter:
+===== Fail2Ban =====
+Add another regex to the fail2ban ''badbots'' filter:
 <file>
+vi /etc/fail2ban/filter.d/apache-badbots.conf
 failregex = ^<HOST> -.*"(GET|POST).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$
             ^<HOST> -.*"GET /bot-trap/
+</file>
+Be sure to enable the ''apache-badbots'' stanza in ''/etc/fail2ban/jail.local'' and restart Fail2Ban:
+<file>
+service fail2ban restart
+</file>
+===== Test Fail2Ban Filter =====
+Modify your log path as necessary:
+<file>
+fail2ban-regex ../logs/access_log /etc/fail2ban/filter.d/apache-badbots.conf
+</file>
+Check the Fail2Ban log:
+<file>
+tail -f /var/log/fail2ban.log
+</file>
+:!: If Fail2Ban fails to parse your log files at all, try setting ''backend=polling'' in ''jail.local''.
+===== Parse IPTables Rules for List of Banned IPs =====
+<file>
+iptables -nL |grep "DROP       all" |tr -s ' ' | cut -d " " -f4|grep -v '0.0.0.0/0' |uniq |sort -n > botlist.txt
 </file>

Virtual Architects Support Wiki

User Tools

Site Tools

Differences

Page Tools