I have customised fail2ban to somewhat passively perform an nmap on the scanning host and email the results to me.

/etc/fail2ban/actions.d/mail-nmap.conf

actionban = nmap -sS -O <ip>|mail -s "[Fail2Ban]: nmap of <ip>" <dest>

/etc/fail2ban/jail.conf

[ssh-iptables]
enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
	   mail-nmap[name=SSH, dest=email@example.com]
logpath  = /var/log/auth.log

One could easily make this somewhat more aggressive. Some ideas I've thought of:

Reflect all traffic back to themselves:

actionban = iptables -t nat -I PREROUTING -s <ip> -j DNAT --to <ip>
	   iptables -I FORWARD -s <ip> -d <ip> -j ACCEPT
actionunban = iptables -t nat -D PREROUTING -s <ip> -j DNAT --to <ip>
	   iptables -D FORWARD -s <ip> -d <ip> -j ACCEPT

Return the favour with your own brute-force attack (I don't recommend this; it's for education purposes only!)

actionban = python brutessh.py -h <ip> -u root -d dictionary.txt

Anyone have any other tricks for custom fail2ban actions?