SSH Tips and Tricks Part 2 - Host Keys
The very first time you log into a new host you will be asked to verify that the system is the one you intended to access. The remote system will send its host key to your client as part of their handshake and your client will ask you to verify the host key fingerprint before continuing the login process. It sounds complicated, but you don’t have to worry about it! Your ssh client does all the work! Here’s an example of what you should see on that first attempt (this will vary based on your OS).
The authenticity of host '[example.com]:22 ([10.2.139.23]:22)' can't be established. RSA key fingerprint is 8c:5b:5e:69:a2:29:98:f5:7d:4a:9a:d3:4c:fe:3f:43. Are you sure you want to continue connecting (yes/no)?
If you answer no, the login process stops and you are given a simple error message. If you answer yes, the login process continues and your client will acknowledge the host and after a brisk handshake the client will store the host key locally in your ~/.ssh/known_hosts file if the file exists or create one for you if it doesn't. Upon accepting the key you will see a warning which might look odd since you meant for this to happen but it’s really just a notice acknowledging the acceptance of the host key.
Warning: Permanently added 'example.com,10.2.139.23' (RSA) to the list of known hosts.
Now you will be asked to enter your password. Enter it and hit return and you should be granted access to the system.
$ firstname.lastname@example.org password:
After this initial storing of the host key you’ll just need to enter your password to login and you will not be asked to verify the key again. In general this means smooth sailing for you.
Host Key Errors
One of the most common errors you will likely encounter is a host key error which may not happen until long after you have made your first connection to a host. If any of several identifying features of the host change a new host key could be created and if that happens your ssh client will let you know by refusing to log you into that system. This is to protect you from “man in the middle” and similar attacks where an attacker tries to insert themselves between you and your desired host. If your client thinks something is amiss you will see a warning similar to the following but if you see this warning don’t panic. There are some valid reasons why the host key may change, such as a hardware upgrade, software updates or even a network change on the host could trigger a key change.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 5c:4b:26:56:b6:c1:21:11:3a:cd:1b:a2:91:cd:f5:1c. Please contact your system administrator. Add correct host key in /home/john.doe/.ssh/known_hosts to get rid of this message. Offending key in /home/john.doe/.ssh/known_hosts:3 RSA host key for example.com has changed and you have requested strict checking. Host key verification failed.
If you do run into a host key identification warning, what should you do? Some ssh clients may ask you right after the warning if you want to continue and skip the host key verification step altogether (which can also be done by passing in the appropriate option to ssh). As I mentioned though, this is a safeguard meant for your protection and should not normally be circumvented.
Before jumping ahead and skipping the verification or messing with your known_hosts file you really should contact the administrator for the host system and verify that the key has indeed changed for a valid reason. They should also be able to provide you with the fingerprint for their new host key so you can compare it to the key your client displayed in the identification error message.
The fingerprint for the RSA key sent by the remote host is 5c:4b:26:56:b6:c1:21:11:3a:cd:1b:a2:91:cd:f5:1c.
This is the fingerprint from the host key being sent to your client and it should match the fingerprint the sysadmin you were just chatting with gave you. If it doesn’t it’s time to start getting suspicious. If you do suspect foul play discuss the issue with the host admin as soon as possible.
Hopefully at this point you have reasonable certainty that the error isn’t related to something malicious so you can get back to correcting the offending key your client has been warning you about. Currently the client trusts the key it has stored and not this new suspect key being presented by the host. You will need to remove the old host key before you can do much else, basically erasing the clients memory of the old key as if it never existed. If your ssh client gives you an option to update the key from the command prompt you can do so there. If not, you will have to use an ssh utility to remove the old key or manually remove it.
Enter ssh-keygen to (hopefully) save the day. ssh-keygen is an ssh utility and has a number of useful functions. Here, running ssh-kegen with the “-R” option allows us to “remove” a host key and specifying the hostname or IP address you were connecting to will remove that specific key from your known_hosts file. As an added perk on most systems, ssh-kegen will create a backup known_hosts file with the removed key(s).
Here's an example of removing our offending host key. You should receive some validation that the key was removed and if you need to you can open the known_hosts file and take a look to be certain.
$ ssh-keygen -R example.com /home/john.doe/.ssh/known_hosts updated. Original contents retained as /home/john.doe/.ssh/known_hosts.old
If you are like me, however, and tend to manually edit the known_hosts file, or if the permissions of the file aren’t perfect (ssh is very picky for your protection) you may end up with errors similar to the following.
$ ssh-keygen -R example.com key_read: uudecode AAAAB3NzaC1yc2EAAAABIwAAAQEApcuBlsCs9dIipb2LHlq9S7nx 1VZcl+UyVc4QMeYmPzu12DSQFpaPVc6eBAgXypyzStUljNg7qsGv4keiJqYltjsB/NWxpl0 3JpbXWS40uoNm7LFYh5HftQ4K0fTZzfpFh2NNkesY2Pq3CrGNDgs6XWLaBPN+W6HTyh5z3w 3Sls/wlyVcobEul2i8KyLy67b6l//RwRpS3Po79Y4r48k9IqpaXEAhycXquETFNLQyXZDkK wZkUjIJc4H62CS8JICW2AHp20Fj/GKSg3grkXZ/ypXed36zckY6vfuQLnzXqUeNBGdFOpP+ E9pefF17YtP7mA/0CNoO0wdSuLbvfjRHw== failed
line 5 invalid key: |1|HaBb84D3PtvhT8dBIiMlwOrJkHY=|GyqqrC7g... /home/john.doe/.ssh/known_hosts is not a valid known_hosts file. Not replacing existing known_hosts file because of errors
No worries though because you can still manually edit the known_hosts file. Review the error message again and note the following line.
Offending key in /home/john.doe/.ssh/known_hosts:3
This shows you that the host key in question is on line 3 in the known_hosts file so you know which one you need to remove. You can just open the file in your favorite dev editor and remove the entire line and resave it. Just deleting the line will force ssh to once again ask you to verify the new host key when you try to log into that host again. Be aware an issue with manual editing of the file is that many text editors have a default character encoding that will cause errors which is why I mentioned "dev editor" for this step.
If all else fails you can delete the known_hosts file completely and start fresh. This actually takes care of many random ssh connection issues at times and occasionally it’s just a good idea to clean house. If you decide to delete the file just move it out of your ~/.ssh directory so you can reference it until your issues are resolved. SSH will happily create a new known_hosts file the next time you try to login with ssh. Remember, you will have erased all of its knowledge about any of the hosts you may have visited in the past, so it’s back to making those handshakes again for each system you visit.
For my next installment I’ll cover ssh key creation and usage. That will be the first power trick you will want to understand since it’s becoming more and more common for systems to be restricted to key only access. Even third party services such as version control hosting will insist that you have a ssh key.