SSH Keys Kinda Suck

OK, I admit to engaging in some attention-grabbing in that heading, and the title. It might even be a little bit unfair, although not much. How about this instead?

SSH Keys Don't Really Improve Security

Ah, maybe even that is a bit much. How about ...

SSH Keys Are About Equal To SSH Passwords In Terms Of Security If You're At All Careful

Better? That's at least not insane sounding, yeah? By "at all careful" there, I mean that you're running denyhosts or fail2ban or checking your shadow file with john or otherwise mitigating against the constant flood of external password-based brute force attempts that's part of having ssh open on the modern internet.

So, on to backing up that claim.

The first thing to understand is that security analysis always starts with attack vectors; if you can't talk about who might attack you, how, and in some cases even why, you can't talk about how to best defend against attack.

SSH-Based Attack Vectors In The Real World

(There's a much shorter version of this below.)

Most sysadmins, at one time or another, have experienced intrusions, successful or otherwise. I am in the unfortunate position of having experience two successful SSH intrusions, which are largely the basis for this essay, because the things that sysadmins are telling each other about how to prevent break-ins simply would not, and could not, have prevented these. In one case, I actually deliberately decided to not do anything to solve the "problem" at all, because the system was working as I had designed it. But I digress.

If you're a sysadmin with any experience, think back to the last time a system of yours was attacked, or even compromised. Or the time before that. Or the time before that. And, as a favour to me, ask yourself this question: when was the last time that a successful (relevantly successful; actual damage and stuff) or near-successful break-in actually had anything to do with anyone's password?

In my case, never, but it does happen of course; there are giant networks of ssh-brute-forcing bots (which, by the way, is what fail2ban is for); presumably it works or they wouldn't bother. It's sill the case that your passwords need to be moronically simple to be vulnerable to this, though, and I don't think that anyone with such passwords on their system would be reading this essay in the first place. Running john against your shadow file every once in a while can't hurt, though.

In my case, I've had a number of serious break-ins or attempts based on web vectors; all easily controllable. I've had two SSH-based attacks that worked: one was a password sniffing sshd installed on another machine, and the other was ssh key based. I get hundreds (literally) of ssh attacks every day; piles of syslog lines about people trying to log in as root or whatever. None of those attacks has ever succeeded at getting in.

The reason for this is very simple: at this time in computer security, for normal people that aren't active targets (try working for Symantec if you want some practice being an active target, BTW), there are exactly two kinds of attacks:

1. Disgruntled employees, who are on site and have legitimate access, designed to trash things or copy them to sell to other companies (most successful attacks are disgruntled employees, by the way; never forget that)
2. Remote attacks over the Internet where you are simply one vulnerable host out of millions, you get added to a botnet and no human ever directly interacts with your machine; if your system doesn't crack on the first few tries, such attackers will just try another host. There's no shortage of vulnerable hosts on the 'net.

The first item there is totally outside the scope of this document, because it is impossible (by definition) to create a purely technical solution to prevent someone from mis-using access that they legitimately possess. If a basic part of a person's job is to delete files under /foo/bar, and they decide to delete everything under /foo/bar because they're mad, no software will stop them; that's an AI complete problem. Now, you can make it harder for them to also delete things under /foo/baz, which they shouldn't be deleting things under, but that's an SE Linux problem or something; it's certainly not an SSH problem, in most cases, and for those cases where it is, everything in this essay applies.

What you do not see on that list of attacks:

1. Individual, targeted attacks that involve a human being selecting you personally, interacting with your systems personally, and being willing to devote anything more than totally insignificant levels computing resources to get in to your machine

Unless you're a top-secret military project, no-one is going to rifle through your garbage looking for your passwords. No-one is going to go wandering around your office looking for passwords. Basic things like 2-second-per-try timeouts on SSH and such make external brute-force attacks a giant waste of computing resources: normal attackers, in real-life, on today's Internet, simply won't bother. They'll move on to easier prey.

If your important passwords are vulnerable to one or two or 50 hits by an external brute-force attack, you fail. If your important passwords are vulnerable to someone getting a copy of shadow and running jack the ripper on it, you also fail, but I've never actually seen this happen, ever, and I've been a sysadmin for almost 15 years now (as of January 2010). If one of your users has such a password, that shouldn't be a big deal either: a user with such a password didn't have any special privileges (RIGHT!?), and non-privileged single-user intrusions are trivial to deal with, in general.

I've never seen this happen either, by the way, via SSH. The 2 second (or whatever) delay on repeat attacks means that basically anything except "password" and password == username is impossible to brute force in a reasonable length of time, and attackers will move on to easier prey.

Attack Vectors: The Summary

The 99.999% (five nines) attack vector on a normal (not government, not a computer security firm, not a super-giant corporation, not a bank) Internet-accessible host on today's modern Internet is an attack of opportunity, delivered in such a way that whether or not your SSH passwords are secure is very close to totally irrelevant.

A Digression: Sticky Notes Are Better Than Simple Passwords

Even when passwords matter, again, unless you're a government or a megacorp, no-one is trolling your office or your garbage for passwords.

There's a lot of back and forth between sysadmins on the questions of: "Which is better? fido21 as a password, memorized, or hncroNTHS31cgrl as a password written down?". Consistently, though, over the last 5 years or so, I've seen sysadmins coming down on the side of the latter. Given the nature of modern computer attacks, I'm going to be public about it:

Unless your situation is very exceptional, a decently secure password written down is better than a weak one memorized. The former is essentially invulnerable to remote brute-force attacks, and that makes all the difference.

Having said that, pass phrases are better than any of that; see http://www.codinghorror.com/blog/archives/000342.html and http://www1.umn.edu/oit/security/OIT74291_REGION1.html#passwords_and_passphrases and other places google knows about.

Another Digression: Remote Fun With sudo

Having password requirements on sudo does not protect you from ssh-key-only attacks. sudo generally has a 5 minute window where users do not need to re-enter the password. An attacker with access to a user's SSH key only needs to run a quiet loop somewhere trying out sudo access and wait until the user legitimately uses sudo. The one time I've been seriously rooted, a totally unrelated machine was compromised, a user with sudo on my box had their SSH key stolen, the SSH key was used to get into my box, and then sudo was run. We don't know if the sudo timing trick I just mentioned was used, or if the user's actual password was sniffed on the compromised machine, but the fact remains: it is possible for an attacker to gain access to a user's sudo privileges ''without ever knowing their password'' if you allow SSH key access.

You can mostly fix this problem this with tty_tickets, though. It's on by default in some distros.

Thanks For Security 101 There; How Do SSH Keys Fit In?

That's simple: in the modern attack environment, SSH keys are a fuzzy, comfy, pretty doormat that says "Please Rape Me".

To understand where I'm coming from here, think about this: when you trust a user's SSH key, what, exactly, are you trusting?

Please actually think about that for a minute.

The answer is: you are trusting the security of every other host that the user with that key ever uses. You are not trusting the user in question, which is the big mistake people make when they think about SSH keys. You're not trusting the user, you're trusting the system where the key is stored. Or, more likely, the systems. Perhaps many of them.

But SSH Keys Have Passwords!

No, they don't. Not from the server's perspective.

Thinking About Security From A Server-Based Perspective

This is the big problem with discussions about ssh keys: people talk about setting up ssh keys from the perspective of the user. As an SSH user trying to ensure that only you get to log in as you, SSH keys with passwords set on them are pretty good, actually. You, of course, engage only in impeccable security practices. You would never have a passwordless key for scripts or backups (I certainly do, by the way, and it's a key that gives root access, but it's associated with a forced command so that's not really a serious issue), and since it's for backups it has to be a root key), you only keep keys on one system (I copy my keys around) that you have personal physical control over (hah!) and you never run ssh-agent (see below). That's great; good for you. Access to your account is now pretty excellently secure.

From the perspective of the server admin, though, none of this is in any way relevant. SSH keys are sent to the server after the password has been used to unlock them. An SSH server has no way to tell if an SSH key has a password set on it.

People have told me, many times, the exact opposite, so this bears repeating:

An SSH server cannot tell if an SSH key used to log in to it has a password set on it or not.

People get this sort of thing really, profoundly, staggeringly wrong. I apologize to the author for picking on a particular article, but this article on "two-factor security" in SSH really upset me, because from a server perspective it's basically a flat-out pack of lies. The author lists a staggeringly complicated set of steps designed to make it so that the user can't log in without using a physical USB key. It's really pretty complicated for a security measure. Here's the fun part, though: the remote SSH server can't even tell that these steps have been done. From the server's perspective, nothing has changed. How can you call it two-factor authentication if the server can't refuse entry unless both factors are present? How can you call it two-factor authentication when the server cannot possibly even notice that one of the factors exists? I don't think you can, and I don't think you should use that term for a setup like this; I find it dishonest, although I'm sure that wasn't the author's intent. This sort of user-centric thinking is endemic to discussions about SSH.

No, Really: SSH Keys Don't Have Passwords (From The Server's Perspective)

If you don't believe me, run sshd with -dddddd or something, and see if you can tell any difference between an ssh key with a password, and the same one without (use ssh-keygen -p to change the password). You can't see a difference because there isn't one: the password is used to decrypt the key on the client end, and only then is the key used to talk to the server.

So, as the admin of a server with lots of different users, you have no way to tell if users have passwords on their ssh keys, and no way to enforce it one way or the other. This means that prudence dictates that you must assume that every ssh key with access to your box has no password.

Even if you trust all your users to have passwords on their ssh keys, though, you still have to assume that, because of ssh-agent.

ssh-agent And Remote Compromises

Just about everyone who has to do ssh more than once a day runs ssh-agent. Why? To save having to type in passwords all the time. From a server security perspective, that's a big red flag: how can ssh-agent do that? Does it store the password in RAM? No, it simply uses the password to decrypt the key and stores the key in RAM.

A machine with ssh-agent running that has been root compromised has no passwords on any key loaded into any ssh agent: anyone with root can just manually attach to a running ssh-agent. There's nothing to prevent this, at all.

In a way, that's irrelevant, though: someone with root on a box your users log in to your machine from is going to have access to your box in short order no matter what, if they want it. There are all sorts of ways to get keys from a compromised box. ssh-agent makes it slightly faster and less risky for the attacker, but it's a fairly minor point. SSH keys and passwords are about equal here, which is my point: SSH keys are not a security panacea.

I'm not talking about the machine a user runs ssh to, I'm talking about the machine they're coming from, where ssh-agent is actually running.

Note that, again, there is no way to tell if a user is using ssh-agent: from the server perspective, a key is a key is a key. Password, no password, ssh-agent, doesn't matter: it all looks the same to the server.

So, ssh-agent means that if the remote machine is compromised, all the keys are too. So we're back to this:

In Summary: Do You Trust Every Machine Every User Logs In To Your Machine From?

Every SSH key your system accepts as solely sufficient for authentication means trusting every system that user might ever decide to log in to your system from. You trust the physical security of that system, and the administrators of that system. You trust the administrators of that system not so much to not be malicious as to be perfectly competent: to stay on top of every security patch forever.

This is also true for passwords via SSH. Pretty much everything I've said is equally true for passwords and keys; that's really the point.

Since you have no way of knowing whether a key used to access your system is password protected (and since people use ssh-agent), you must assume none of them are password protected.

If you really do trust all of those systems, great; you're far luckier than I am.

OK, So What Do We Do?

Well, for starters, you could stop telling me how insecure passwords are with SSH. As far as I can tell, in the real world, SSH keys are far more insecure than SSH passwords. With SSH passwords, I only need to trust the user, and the physical environment (sticky notes the password is on in the user's wallet, or whatever; I'm assuming that no user with actual privileged access is stupid enough to write the password down in plain text on some remote computer). With SSH keys, I have to trust every computer system the user might ever decide to log in from, and in some cases the systems they might log in to those systems from too. That's a lot more things to trust, and given the nature of modern attacks, I am far more worried about compromises to random systems on the 'net than I am about what's in a user's wallet.

I'm of the opinion that ssh keys and passwords are about equal in terms of security; they both have their places. Neither is a panacea, and you are not committing security suicide by leaving either enabled.

The right solution for secure systems, IMO, is the RequiredAuthentications keyword that the actual SSH Communications Security company's SSH implementation (but not openssh) supports. Require both a key and a password; that'll clear the problem right up. Casual ssh-key-based attacks coming from some other random machine that got compromised and had one of your user's keys on it become a thing of the past.

What's that, you say? That's horribly inconvenient, because now the user has to type in a password?

Welcome to discussions of security. There is almost always a trade off between convenience and security.

It doesn't matter anyway, because the sshd on all the systems I use doesn't have that capability anyways. I would like to see that fixed some day, though; for systems where I really care about security, ssh key attacks scare me quite a bit, having already been bitten by them once now.

Exceptions

There are a couple of cases where it really matters whether you're using keys or passwords

Keyboard Logging

As far as ssh keys being obviously more insecure than passwords, trojans that try to capture the user typing passwords can level the playing field there. I consider this something of a red herring, for several reasons:

1. In all the attacks and near-attacks I've seen, I've never actually seen a keyboard logger, or anything like it. But then, this was all on Linux machines (I use Windows for my desktop, but it's not accessible from the general Internet).
2. While there are cases where it's possible for a trojan to know "Hey, this string right here is a password" (such as a password dialog), I'm not aware of any usage of SSH software on Windows (which is the attack vector we're talking about: most other OSes don't really have key logging trojans) where such a trick works (although if you store your passwords in the SSH program's config, that's probably vulnerable). This means a human has to sift through the key log, which is a non-casual attacker and not what I'm worried about as I've said.
3. Most importantly of all, though: even if keylogging trojans were a pervasive problem that didn't require human intervention, this wouldn't make ssh keys more secure than passwords, it would just make the two exactly the same for my purposes: I can't trust any machine a user logs in from, in either case.

Scripts

Scripts that need remote access should always use keys, and the keys should be linked to a forced command and a particular incoming IP on the remote end. Having a password in a script is inexcusable. See "man authorized_keys", in particular the command= and from= options.

---