Fail2Ban is a service that scans log files for event such as failed login attempts and then updates firewall rules to ban connections from that address. This doesn’t solve problems with weak authentication but it does greatly slow down the rate of attacks. Fail2Ban is a must have if you run an accessible SSH server.
Installation
Start off by making sure a firewall is installed. After much work I was able to get nftables working but it was a steep learning curve. ufw might be a simpler option but now I’ve got nftables working I’m not switching.
Installing Fail2Ban on Debian is simple:
sudo apt update && apt upgrade sudo apt install fail2ban
After installation if you run the command:
systemctl list-units -a --state=active --type=service
You should see fail2ban listed and marked as active. You can check the Fail2Ban service specifically with:
systemctl status fail2ban.service
Sticking a sudo in front of that command will give you log records too.
Configuration
The configuration for fail2ban is all held in the /etc/fail2ban directory and it’s sub-directories. Start off by having a read of the jail.conf file. As recommended in the file, I won’t be modifying it, but it’s good to know what the defaults are. Most guides at this point tell you to take a copy of the jail.conf file and call it jail.local but this is wrong according to the man page. The man page suggests that only settings that are different to the default values should be contained in the jail.local file. This makes perfect sense, why duplicate the entire base configuration file? So switch to the fail2ban configuration directory and open a new file called jail.local.
cd /etc/fail2ban sudo nano jail.local
Avoiding a Self-Ban
I’ll start by adding a default section so that I can override some of the default settings. In particular I want to add an always allow for my IP as I don’t want to accidentality ban myself. The settings for that look like this.
[DEFAULT] ignoreip = w.x.y.z
What action takes place when banning?
What Fail2Ban does when it wants to ban an IP is controlled by the action setting. This can be specified at a per-service level but most of the time you’ll probably want to rely on the default setting. The default setting, as shipped, is action = %(action_)s which is just a pointer to another setting which is this:
action_ = %(banaction)s[port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
This setting then points to another setting called banaction which is this:
banaction = iptables-multiport
The banaction setting tells fail2ban to use the iptables-multiport.conf to perform a ban. This configuration file points to the iptables.conf file and has a note indicating it’s been superseded, I assume this configuration file will vanish in a future release. The iptables.conf file, however, will update the firewall to ban the IP address – but see below.
While the action_ configuration is probably fine for most people there are other versions and you could also define your own. For example action_mw will send an email containing whois information.
Switching from IPTables to nftables
The default firewall on Debian is nftables rather than iptables. Fortunately for us Fail2Ban supports both systems (it also supports ufw which is what I might end up using if nftables proves to hard for me to understand). Add the following setting under the default section in jail.local to switch to nftables.
banaction=nftables[type=multiport]
Turn on a Jail
When you install fail2ban on Debian you get a file called defaults-debian.conf in the /etc/fail2ban/jail.d directory. The file only contains two lines to enable a jail for sshd. The settings in the default file are fine and will be overridden in by settings in jail.local but I don’t particuarly like having multiple levels of override if I can avoid it, it just makes reasoning about the settings more complicated. For that reason I delete this file and instead add the following to jail.local:
[sshd] enabled = true
This simply turns on the jail for sshd (and is exactly what was found in the package supplied file).
Further sshd Settings
The default settings for sshd found in jail.conf are as follows:
[sshd] port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s
The port setting tells fail2ban what port sshd is listening on. If you move the service to a different port then you’ll need to update this in jail.local. Logpath indicates where the log file for sshd is found. Right at the top of the jail.conf file is a setting before = paths-debian.conf which loads a configuration file containing paths for various pre-defined services. The Debian package loads a Debian specific file first and then the paths-common.conf file.
The last line sets the backend that should be used for sshd scanning. The backend is how fail2ban finds log records to parse to find IP’s to ban. The default setting in jail.conf is backed = auto which causes it to use pyinotify, gamn and polling in that order – essentially three different ways of checking log files for changes. The default setting for sshd is backend = %(sshd_backend)s which references a variable in paths.common.conf which in turn references the default setting.
Nobody is Getting Banned! Using the SystemD Journal
If like me you are using Debian 12 then you’ll almost certainly find that no bans are taking place even though your sshd service is getting attacked left, right, and centre. This is almost certainly because Fail2Ban is scanning auth.log and Debian 12 has switched over to using systemd-journald. The traditional log files such as syslog, messages, and auth.log no longer exist (well, I have an auth.log file but it’s empty). To make Fail2Ban use these logs you need to change the backend to systemd. You can either enable the systemd backend as the default or set it just for the sshd service. The issue with setting systemd as the global default is that it’s always present so Fail2Ban can’t detect a missing log file and might silently fail to work (this is discussed in this bug report). For now I plan on setting systemd only for sshd. The sshd section in jail.local is shown below :
[sshd] enabled = true backend=systemd
Being a Little More Harsh
I feel the default settings for Fail2Ban are quite generous to the individuals that are attacking our systems. By default a ban will only be issued if there are five failed attempts in ten minutes and the ban is only ten minutes. I feel default settings along the lines of those shown below are more fitting, obviously they can be changed on a per-service basis if they are too harsh for one particular service:
bantime = 60m findtime = 60m maxretry = 2
Restart Fail2Ban
Once you’ve finished modifying the configuration files you need to restart Fail2Ban:
sudo systemctl restart fail2ban
Completed Configuration
The completed configuration files jails.local will look something like the one shown below. That’s all the configuration you need.
[DEFAULT] ignoreip = 104.27.157.21 bantime = 60m findtime = 60m maxretry = 2 banaction=nftables[type=multiport] [sshd] enabled = true backend=systemd
Monitoring Bans
To quickly see how many IP addresses have been banned you can run the command shown below which will also give you details of what IP addresses are banned (this is real data):
$ sudo fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 4 | |- Total failed: 4 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 8 |- Total banned: 8 `- Banned IP list: 168.232.79.91 23.137.200.58 27.254.235.3 8.219.247.130 8.219.250.105 8.222.169.102 8.222.173.181 8.222.175.161
If for some reason you want to unban an IP you issue this command (replacing the IP as required):
sudo fail2ban-client set sshd unbanip w.x.y.z
If you want to see what Fail2Ban is doing and check for issues tail it’s log file:
sudo tail -n 200 /var/log/fail2ban.log
Fail2Ban Isn’t Working
Before you call the job done you should take a look in the Fail2Ban log file at /var/log/fail2ban.log. If you see a number of error messages like the ones shown below (trimmed for readability) then you don’t have a firewall installed. By default a minimal Debian install doesn’t come with a firewall configured. Additionally, Debian installs nftables rather than iptables and at the point where I captured this error message I hadn’t switched the configuration for Fail2Ban.
NOTICE [sshd] Ban 14.63.196.175
2024-10-20 11:23:41,422 fail2ban.utils [52335]: ERROR 7f64f6334cb0 -- exec: { iptables ... snip ...
2024-10-20 11:23:41,422 fail2ban.utils [52335]: ERROR 7f64f6334cb0 -- stderr: '/bin/sh: 1: iptables: not found'
2024-10-20 11:23:41,422 fail2ban.utils [52335]: ERROR 7f64f6334cb0 -- stderr: '/bin/sh: 1: iptables: not found'
2024-10-20 11:23:41,422 fail2ban.utils [52335]: ERROR 7f64f6334cb0 -- stderr: '/bin/sh: 3: iptables: not found'
2024-10-20 11:23:41,422 fail2ban.utils [52335]: ERROR 7f64f6334cb0 -- returned 127
2024-10-20 11:23:41,422 fail2ban.utils [52335]: INFO HINT on 127: "Command not found". Make sure that all commands in ...snip ...
2024-10-20 11:23:41,422 fail2ban.actions [52335]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' ... snip...
Digging Deeper
At this point you should have a working Fail2Ban system set up, this section is just for interest and looks a little deeper into how Fail2Ban works. Fail2Ban is conceptually quite simple. It runs a set of regular expressions against log files picking out the IP addresses of machines attacking your server. It then updates the firewall to ban those IP addresses from making a connection.
If you want to really see what Fail2Ban is doing you need to look at the firewall itself. The firewall is provided by nftables and can be quite complex but we can certainly dip just a toe in to have a look at what is going on. We’ll start, however, by looking to see what is going on it Fail2Ban:
$ sudo fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 1 | |- Total failed: 1 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 1 |- Total banned: 1 `- Banned IP list: 209.141.43.197
As you can see Fail2Ban has found an IP address that it thinks should be banned. The low count for total failed is due to the fact I’ve only just restarted Fail2Ban for this article. If we now look at the firewall we can see that Fail2Ban has added a table, chain and set for itself.
$ sudo nft list ruleset
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
... snip ....
}
chain forward {
type filter hook forward priority filter; policy drop;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
table inet f2b-table {
set addr-set-sshd {
type ipv4_addr
elements = { 209.141.43.197 }
}
chain f2b-chain {
type filter hook input priority filter - 1; policy accept;
tcp dport 22 ip saddr @addr-set-sshd reject with icmp port-unreachable
}
}
This configuration really shows the power of nftables. Fail2Ban just needs to add the IP addresses it wants to ban to the addr-set-sshd set and the rule referencing the set will do the rest. If you look a the settings for the f2b-chain it’s hooked into input, same as our main chain, but it’s set with a priority of filter -1. Lower priority numbers run earlier which means Fail2Ban will get a chance to reject packets before they get anywhere near our main firewall rules, they can never see the accept rule.
Restarting the Firewall
When you restart the firewall it reads /etc/nftables.conf which probably doesn’t contain a configuration for the Fail2Ban tables. Looking at the documentation it seems that you can specify a table name, etc for Fail2Ban to use but I honestly don’t think it’s necessary. If Fail2Ban doesn’t find the table, chain, etc that it’s expecting it creates it. This causes some nasty looking error messages (below) in the Fail2Ban log but otherwise seems harmless enough. I’ve watched it recreate the table and start populating it again. The only downside I can see, and I admit I’ve not carefully checked this yet, is that you’ll let people out of jail too early as you are causing the set of banned IP addresses to be removed.
The first error message is caused by it not being able to find the set of banned sshd IP address. The second error message, after the snip, is there because it tried to remove a banned address that it couldn’t find – the firewall restart emptied the set of banned addresses. The error messages will stop appearing once Fail2Ban has released all the addresses it has in jail.
[59826]: ERROR 7fb261b6abb0 -- exec: nft add element inet f2b-table addr-set-sshd \{ 47.236.125.178 \}
[59826]: ERROR 7fb261b6abb0 -- stderr: 'Error: No such file or directory'
[59826]: ERROR 7fb261b6abb0 -- stderr: 'add element inet f2b-table addr-set-sshd { 47.236.125.178 }'
[59826]: ERROR 7fb261b6abb0 -- stderr: ' ^^^^^^^^^'
[59826]: ERROR 7fb261b6abb0 -- returned 1
... snip ...
[59826]: NOTICE [sshd] Unban 185.233.36.199
[59826]: ERROR 7fb261b6a8b0 -- exec: nft delete element inet f2b-table addr-set-sshd \{ 185.233.36.199 \}
[59826]: ERROR 7fb261b6a8b0 -- stderr: 'Error: Could not process rule: No such file or directory'
[59826]: ERROR 7fb261b6a8b0 -- stderr: 'delete element inet f2b-table addr-set-sshd { 185.233.36.199 }'
[59826]: ERROR 7fb261b6a8b0 -- stderr: ' ^^^^^^^^^^^^^^'
[59826]: ERROR 7fb261b6a8b0 -- returned 1
Why not just create the table, chain, etc in /etc/nftables.conf? You could but then you have to maintain the configuration in two places. Changes in Fail2Ban need to be mirrored in the nftables configuration. In particular the sets and rules need to be created on a per-service basis. Overall I feel it’s simpler to just let Fail2Ban manage it on it’s own. Additionally, it’s not clear to me how you’d maintain the list of banned IP addresses so most of the error messages will appear anyway.
Note: it seems that just restarting Fail2Ban might recreate the table and reinstate all the bans. I believe it’s possible to make one service restarting trigger another to restart so there’s probably a way of doing this automatically, maybe one day I’ll look into that…
References
- Fail2Ban website and Github
- A surprisingly good guide for Debian 11
- A comprehensive guide but with little explanation
- Logs in Debian 12
- Making nftables and fail2ban work together – this is a good page but some of it is no longer relevant and there’s a mistake regarding priorities.