User Tools
networking:linux:bad_bot_trap
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
networking:linux:bad_bot_trap [2011/11/09 14:50] gcooper |
networking:linux:bad_bot_trap [2013/10/13 14:59] (current) gcooper |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Bad Bot Trap ====== | ====== Bad Bot Trap ====== | ||
| + | |||
| + | See also **[[networking: | ||
| http:// | http:// | ||
| - | Used here with **[[networking:linux:fail2ban|fail2ban]]**: | + | We use Fail2Ban to block bad bots. |
| + | |||
| + | ===== robots.txt ===== | ||
| + | |||
| + | Misbehaving bots may access areas of your web site even if you tell then not to. That's what we key on here. | ||
| + | |||
| + | :!: First, change to your web root folder. | ||
| + | |||
| + | Edit robots.txt and add a '' | ||
| < | < | ||
| - | vim robots.txt | + | cd public_html |
| + | |||
| + | vi robots.txt | ||
| User-agent: * | User-agent: * | ||
| Line 12: | Line 24: | ||
| </ | </ | ||
| - | < | + | ===== Web Site Header ===== |
| - | mkdir / | + | |
| - | vim /var/www/web1/web/ | + | We use a tiny image for embedding the hidden link. '' |
| + | |||
| + | < | ||
| + | cd images | ||
| + | wget http://www.sonoracomm.com/images/blank.png | ||
| + | cd .. | ||
| </ | </ | ||
| - | Add this in the header. | + | Edit your HTML header |
| < | < | ||
| + | vi index.html | ||
| + | |||
| <a href="/ | <a href="/ | ||
| </ | </ | ||
| - | Now we create a file so as not to pollute the error logs: | + | ===== bot-trap Folder ===== |
| - | < | + | Now we create the '' |
| - | vim / | + | |
| + | < | ||
| + | mkdir bot-trap | ||
| + | cat << EOF >> bot-trap/ | ||
| < | < | ||
| < | < | ||
| Line 36: | Line 56: | ||
| </ | </ | ||
| </ | </ | ||
| + | EOF | ||
| + | |||
| + | chown -R apache.apache bot-trap | ||
| </ | </ | ||
| - | Add another regex to the fail2ban filter: | + | ===== Fail2Ban ===== |
| + | |||
| + | Add another regex to the fail2ban | ||
| < | < | ||
| + | vi / | ||
| + | |||
| failregex = ^< | failregex = ^< | ||
| ^< | ^< | ||
| + | </ | ||
| + | |||
| + | Be sure to enable the '' | ||
| + | |||
| + | < | ||
| + | service fail2ban restart | ||
| + | </ | ||
| + | |||
| + | ===== Test Fail2Ban Filter ===== | ||
| + | |||
| + | Modify your log path as necessary: | ||
| + | |||
| + | < | ||
| + | fail2ban-regex ../ | ||
| + | </ | ||
| + | |||
| + | Check the Fail2Ban log: | ||
| + | |||
| + | < | ||
| + | tail -f / | ||
| + | </ | ||
| + | |||
| + | :!: If Fail2Ban fails to parse your log files at all, try setting '' | ||
| + | |||
| + | ===== Parse IPTables Rules for List of Banned IPs ===== | ||
| + | |||
| + | < | ||
| + | iptables -nL |grep " | ||
| </ | </ | ||
networking/linux/bad_bot_trap.1320875427.txt.gz · Last modified: 2011/11/09 14:50 by gcooper
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
