This post is going to provide a brief description of basic Linux hardening for the Bastion Host I created using CloudFormation in my last post. Really it’s hardening 101 for any Linux host, but this kind of information is surprisingly hard to find on the Internet. For instance, I read a bunch of posts that say don’t run services you don’t need, but with no practical advice on how to implement that (like how to figure out what services are running on a typical Linux box, or which specific ones I might want to stop and why). The absolute best post I’ve seen on the subject is 40 Linux Server Hardening Security Tips [2019 edition]. I didn’t actually find this post until after I’d worked through the things I’m going to implement in this post, but it covered them all and a whole lot more, so you could just go there and skip this post, however, it’s pretty complex and hard to follow in some parts, and I’m going to provide a simple explanation for some simple hardening, so you might want to stick around.
Most of the work will be performed by updating my bootstrap script, with one small change to the CloudFormation template. I’m going to insert all of my changes into the script just before running the yum update, so just before the following lines:
1 2 | # Run system updates yum -y update |
Changing the Port for SSH
The first thing I’m going to do is change the port on which SSH is listening. Let’s not kid ourselves, this isn’t really security, but it will cut down on some of the automated bot scanning for SSH listeners. I’m going to change the port to 443, which is the standard port for SSL web servers. I do this because a lot of companies block outbound connections to port 22, but not to port 443 (it would be pretty hard to block 443 and still have a useful corporate network). Keep in mind that bypassing your company’s firewall by using 443 for SSH may be frowned upon or even violate your company’s security policy, so buyer beware.
So I insert the following lines in my bootstrap script:
1 2 3 | # Move sshd to port 443 sed -i "s/#Port 22/Port 443/" /etc/ssh/sshd_config systemctl restart sshd |
I’m just using sed to search and substitute in the configuration file /etc/ssh/sshd_config, to replace the text “#Port 22” with “Port 443”. Then I use systemctl to restart the SSH service.
This is the one change I’m going to make that requires an update to my CloudFormation template too. I need to update my security group so that it allows inbound port 443 instead of port 22. Here is that change:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | "BastionSecurityGroup": { "Condition": "CreateSecurityGroup", "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "Enable TCP on port 443 (SSH)", "VpcId": { "Ref": "VpcId" }, "SecurityGroupIngress": [{ "CidrIp": { "Ref": "SshSourceCidr" }, "IpProtocol": "tcp", "FromPort": 443, "ToPort": 443 }], "Tags": [{ "Key": "Name", "Value": "BastionSG" }, { "Key": "StackName", "Value": { "Fn::Sub": "${AWS::StackName}" } }, { "Key": "Network", "Value": "Public" } ] } } |
And that’s that. Once I update my CloudFormation stack, any new Bastion Host that spins up will be listening on port 443 for SSH. Now do remember that my CloudFormation template allows you to specify the id of an existing security group, in which case it will not create one. If you used that option, you will have to make the appropriate changes to the existing security group yourself, or you won’t be able to get to SSH at all.
Installing Packages
Now I need to install a couple of packages that aren’t installed by default on AWS Linux 2, and in fact aren’t in the yum repositories that are normally configured for AWS Linux 2. So first, I’m going to install a package that will configure the yum repository for the Fedora Project’s Extra Packages for Enterprise Linux (or EPEL):
1 2 | # Install the RedHat epel yum repo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm |
This repository contains the other 2 packages I want to install, which I do like so:
1 2 | # Install iptables-service and fail2ban from the epel repo yum -y install iptables-services fail2ban |
This installs the following two packages:
- iptables-services – this installs the systemd scripts to allow controlling iptables as a service using systemctl. These days it appears to be fashionable to firewall Linux using firewalld instead of iptables, but I’m an old dog and like iptables, so I’ll stick with that. Also, fail2ban works with iptables (I believe it also works with firewalld).
- fail2ban – a nifty little package that scans log files and implements rules that can do things like dynamically block an IP address for a configurable period of time if there are more than X number of failed login attempts from that address. This can really slow down brute force attacks, notify you when such attacks are occurring, and give you an opportunity to take corrective action if required.
Linux Hardening: Configuring iptables
iptables is a command-line tool for configuring firewall rules, which are actually enforced by the Linux kernel. It is also a service that can be configured to run a set of firewall rules each time a Linux machine spins up. First, we’re going to enable and start it as a service:
1 2 3 | # Enable iptables to start on boot, and start it now systemctl enable iptables systemctl start iptables |
Now remember that ststemctl on AWS Linux 2 doesn’t know anything about iptables by default. Installing the package iptables-services put the proper configuration files in place so systemctl can work with iptables. We still need to configure some iptables rules, which looks like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # Configure iptables: # 1. accept anything on the loopback adapter # 2. accept incoming packets that belong to a connection that has already been established (using the state module) # 3. accept udp on ports 67:68 (DHCP) # 3. accept tcp on port 443 (where we're running sshd) # 4. drop anything else # and persist the config iptables -F iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A INPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT iptables -A INPUT -j DROP iptables-save > /etc/sysconfig/iptables |
The comment explains what we’re doing line-by-line, but to paraphrase further, I’m allowing any inbound packets on the lookback adapter, any inbound packets that are part of an established connection, inbound DHCP (so we can get our dynamic IP address), and inbound TCP on port 443 (where we’re running SSH). All other inbound packets are silently dropped.
These commands demonstrate running iptables as a command-line utility. This creates a set of in memory rules, meaning if I reboot the box they’ll be wiped clean. The last line actually persists the rules so the iptables service will reinitialize these rules in the kernel on boot. iptables-save just prints to standard out the current in memory rules, and saving this output to the file /etc/sysconfig/iptables tells the iptables service to run these rules on boot.
As you can see, that wasn’t a great deal of work, but we’ve already taken significant strides towards Linux Hardening our Bastion Host.
Linux Hardening: Configuring fail2ban
fail2ban is a service which, once installed, has the needed systemd configuration files to allow systemctl to work with it. So again, we need to configure it to run on startup:
1 2 3 | # Enable fail2ban to start on boot, and start it now systemctl enable fail2ban systemctl start fail2ban |
But of course, we still need to configure it to do something. Now it is installed with a default configuration file in /etc/fail2ban/jail.conf. And this file is already configured to work with SSH, but not on port 443, it expects SSH to be running on the default port of 22. Whatever you do, don’t modify the jail.conf file. This file will almost certainly be overwritten if you upgrade the package. But fail2ban will read it’s configuration from a jail.local file in the same directory, if it exists. So first, you copy the jail.conf file to jail.local. Then modify the jail.local file to your heart’s content. That’s what the following does:
1 2 3 4 5 6 7 8 9 10 | # Configure fail2ban by copying jail.conf to jail.local and: # 1. lower maxretry to 3 in the jail.local file # 2. enable the sshd-iptables in the jail.local file # 3. change the ssh port to 443 in the jail.local file # and restart fail2ban cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local sed -i "s/maxretry = 5/maxretry = 3/" /etc/fail2ban/jail.local sed -i "s/^\[sshd\]/[sshd]\nenabled=true/" /etc/fail2ban/jail.local sed -i "s/port *= *ssh/port = 443/" /etc/fail2ban/jail.local systemctl restart fail2ban |
Again the comments say it all. I lower the number of failed logins to 3 before an address is blocked. I then enable the iptables plug-in for fail2ban. And finally, I change the SSH port to 443. Restart fail2ban and we’re good to go.
To test this, I’ll try to ssh into the bastion on 443 using the wrong user name. The first 3 times, all will go as expected. The fourth time I try, my ssh client will hang, and eventually timeout. I stuck with the default timeout period of 10 minutes, so after this time I can try again without hanging. fail2ban just inserts temporary rules into iptables to block the offending IP address, and after the timeout period it removes them. These are in memory rule that are never persisted, so a reboot should get you back in faster if you don’t want to wait. If you’re logged in, you can use iptables-save to see what these rules look like.
Now a quick note about how fail2ban works, and the limitations of that approach. fail2ban monitors the system log files, and it’s configured with a bunch of log message patterns that will trigger a response. For AWS Linux 2, failed logins are logged to /var/log/secure. If I do a tail -f on that file, I’ll see the logged messages when I test my configuration as described above. But, when I first started messing with this, instead of using an incorrect user name, I tried using an invalid certificate. I did see the messages for the failed authentication appear in the log file, but fail2ban clearly didn’t match that message and I never got blocked. fail2ban is fully configurable, as in I can write my own message patterns to it’s configuration, but it’s a bit complex and I haven’t dug that deep into it yet. Perhaps someday that will be another blog post, but for now it works only as well as fail2ban OOB.
Linux Hardening: Removing Unnecessary Services
As a last step, I’m going to look at what services are running on my spiffy new AWS Linux 2 Bastion Host. Ssh into it (remember to specify the port 443). Then run the command ‘lsof -i -n -P | grep LISTEN’:
The output is a list of listener ports currently open on my machine, and what process is listening.
The first line of the output is for Postfix mail system listening on port 25 (but only on localhost/127.0.0.1). The Postfix mail system is a small SMTP email server implementation, but only listening on localhost and blocked from inbound connections using iptables, I’m not that worried about it. Plus, I may want to configure it later so I can send outbound email for fail2ban alerts, so I’m going to leave that one alone for now.
The next two entries are for SSH on port 443. Note that there are two of them because one of them is for IPv4 and one of them is for IPv6. My host doesn’t have an IPv6 address, so I don’t really need that one and may want to look into removing that listener from the SSH configuration, but that also means it doesn’t present much of a security concern so I’m going to leave that one alone for now too.
The last two entries are for rpcbind. RPC has always been a potential problem from a security point of view, and RPC bind itself is known to be vulnerable to Denial of Service (DOS) attacks, so unless you need it, this service should be stopped and disabled. How do you know if you need it? The most common reason you might is if you were going to be using Network File Service (NFS). If you don’t know if you need it, you probably don’t, particularly on a Bastion Host.
Anyway, to stop rpcbind, type the command ‘systemctl stop rpcbind’, after which if you rerun lsof it should show:
And that looks pretty good. I’m only listening to ports 25 and 443. And port 25 is only bound to localhost and blocked from inbound access by iptables. Fortunately, AWS Linux 2 doesn’t run a bunch of unnecessary services by default. If you’re on some other Linux distribution, you may find a great many other services running by default, in which case you’ll need to do some research on which ones you may want to shutdown from a security point of view.
Now I’ve stopped the service, but I haven’t disabled it. If I reboot it will spin up again. To take care of that, add the following lines to the bootstrap script:
1 2 3 | # Shutdown RPC bind (used for NFS, not need on this server) systemctl stop rpcbind systemctl disable rpcbind |
FYI, if you decide for some reason that you do need to run RPC bind, then you will probably want to relax the iptables rules a bit to allow inbound packets on port 111 from a select list of local IP addresses. RPC bind should never be generally accessible from the Internet (see the references for more info on that).
Sum Up
As you will see if you follow the link to 40 Linux Server Hardening Security Tips [2019 edition], there’s a lot more I could do. It’s just a question of how valuable is your time vs. how valuable are the resources you’re trying to protect. I’m protecting a WordPress blog. I’d rather not get hacked, but it’s not like I’m protecting amazon.com where minutes of downtime can cost many thousands of dollars. Given that, I think I’ve done enough hardening for now.
That said, there is one more thing I will do to further secure my environment. I’m going to go into the AWS console and configure the minimum and desired number of instances for my Bastion scaling group to 0. The scaling group will then power down my Bastion. When I want to do some administration, I’ll configure it back to 1 and it will come back up. From a security point of view, this is a good thing. There’s no way to SSH into my network whenever I’m not performing administration. And I’ll probably mostly be doing patching say once a month. It will also save me money. Why run a bastion 24/7 when I only need it once a month for a few hours max?