Virtual Architects Support Wiki

User Tools

Site Tools

You are here: start » networking » linux » bad_bot_trap
Trace:
networking:linux:bad_bot_trap

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
networking:linux:bad_bot_trap [2013/09/27 11:52]
gcooper 		networking:linux:bad_bot_trap [2013/10/13 14:59] (current)
gcooper
Line 1: Line 1:
 ====== Bad Bot Trap ====== ====== Bad Bot Trap ======
- 
-FIXME 
- 
-===== Reference ===== 
  
 See also **[[networking:linux:fail2ban|Fail2Ban]]** See also **[[networking:linux:fail2ban|Fail2Ban]]**
Line 10: Line 6:
  
 We use Fail2Ban to block bad bots. We use Fail2Ban to block bad bots.
 +
 +===== robots.txt =====
 +
 +Misbehaving bots may access areas of your web site even if you tell then not to.  That's what we key on here.
  
 :!: First, change to your web root folder.  Possibly ''public_html''. :!: First, change to your web root folder.  Possibly ''public_html''.
Line 24: Line 24:
 </file> </file>
  
-<file> +===== Web Site Header =====
-mkdir bot-trap +
-</file>+
  
-''blank.png'' should be just a single pixel transparent image and is unimportant (and may already exist).  You can get ''blank.png'' like this:+We use a tiny image for embedding the hidden link.  ''blank.png'' should be just a single pixel transparent image and is unimportant (and may already exist).  If it doesn't already exist, you can get ''blank.png'' like this:
  
 <file> <file>
Line 36: Line 34:
 </file> </file>
  
-Edit your HTML header (''<head>'' section) (''index.html'', ''templates/yourtemplatename/index.php'', etc.) and add this line:+Edit your HTML header (''<head>'' section) (''index.html'', ''templates/yourtemplatename/index.php'', etc.) and add this line (modify as necessary):
  
 <file> <file>
Line 44: Line 42:
 </file> </file>
  
-Now we create a file so as not to pollute the error logs.  Change the URL to your own domain:+===== bot-trap Folder ===== 
 + 
 +Now we create the ''bot-trap'' folder and a file so as not to pollute the error logs.  Change the URL to your own domain and the permissions as necessary:
  
 <file> <file>
 +mkdir bot-trap
 cat << EOF >> bot-trap/index.html cat << EOF >> bot-trap/index.html
 <html> <html>
Line 56: Line 57:
 </html> </html>
 EOF EOF
 +
 +chown -R apache.apache bot-trap
 </file> </file>
 +
 +===== Fail2Ban =====
  
 Add another regex to the fail2ban ''badbots'' filter: Add another regex to the fail2ban ''badbots'' filter:
  
 <file> <file>
-$ vi /etc/fail2ban/filter.d/apache-badbots.conf+vi /etc/fail2ban/filter.d/apache-badbots.conf
  
 failregex = ^<HOST> -.*"(GET|POST).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$ failregex = ^<HOST> -.*"(GET|POST).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$
-            ^<HOST> -.*"GET /bot-trap/"+            ^<HOST> -.*"GET /bot-trap/
 </file> </file>
  
Line 72: Line 77:
 service fail2ban restart service fail2ban restart
 </file> </file>
 +
 +===== Test Fail2Ban Filter =====
 +
 +Modify your log path as necessary:
 +
 +<file>
 +fail2ban-regex ../logs/access_log /etc/fail2ban/filter.d/apache-badbots.conf
 +</file>
 +
 +Check the Fail2Ban log:
 +
 +<file>
 +tail -f /var/log/fail2ban.log
 +</file>
 +
 +:!: If Fail2Ban fails to parse your log files at all, try setting ''backend=polling'' in ''jail.local''.
  
 ===== Parse IPTables Rules for List of Banned IPs ===== ===== Parse IPTables Rules for List of Banned IPs =====
networking/linux/bad_bot_trap.1380304358.txt.gz · Last modified: 2013/09/27 11:52 by gcooper

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki