Mitigating the Security Risks of SSH

Date: Aug 25, 2006

John Tränkenschuh describes ways to create a solid security plan to lessen the unknown factors of SSH security. To mitigate SSH risks, we don't begin by considering the server or client technical settings; instead, we begin by considering the implementation challenges to organizations.

This article is a follow-up to my earlier article on SSH security considerations, in which I discussed some very real risks with SSH—risks created by how SSH is rolled out in many organizations. Large organizations with many production platforms are especially at risk. Slashdot featured some excellent discussion about that article, along with other responses that convinced me of the need for a follow-up. In this article, we’ll consider mitigations (security controls) that you can apply to the SSH rollout at your organization.

This article will help the UNIX administrator—maybe one who’s new to SSH—to see the need for a better security plan. It will also help the administrator who’s rolling out SSH on a non-UNIX platform. (No one can expect UNIX administrators to own the SSH security plan, even if some consider SSH to be a UNIX utility.)

The True Issues

SSH is a wonderful tool with many excellent abilities. It has versions for OpenVMS, Windows, z/OS, iSeries, UNIX and Linux, etc. Thanks to the work of the commercial and open source SSH vendors, this tool is getting a lot of recognition. Because more and more applications must fan across data repositories living on multiple platforms, people want to use SSH as a more secure replacement for Telnet, FTP, and rlogin/rsh.

Many UNIX administrators will read each SSH man page thoroughly. They appreciate the risks of a tool that moves files, allows remote execution, and can encrypt and tunnel network traffic into any TCP portstream. But what about administrators using SSH on other platforms? Will they just plop in this tool as a simple FTP replacement, get it to work in that limited role, and then declare success?

The biggest issues with SSH lie at Layer 8 of the OSI model—politics and personnel:

One vulnerability issue underlies all SSH implementations: Most administrators know nothing about SSH’s port-forwarding abilities (or choose to ignore them). They may very well regard the security problems as "a UNIX issue." So the first risk is proliferation of a naïve SSH security design across multiple platforms, with little ownership of the big issues.
A second risk is the "convenience at all costs" approach to agent forwarding. Anyone who has read an SSH man page knows that agent forwarding has known risks when used in untrusted environments. Do the same vulnerabilities exist with other operating systems? For that matter, do all client and server SSH implementations carry the same warnings? We can’t answer all of these questions, but we can make a strong recommendation and review a suggested Slashdot poster’s mitigation.
A third major issue is the port-forwarding risk that allows an innocent outbound connection (to a remote SSH server) to become a malicious inbound connection into your company’s intranet. This connection is encrypted and will be very difficult to monitor, thus adding to the danger.

Security mitigations must do more than suggest technical settings for one SSH version. (And the technical settings vary by version, anyway, so don’t expect this article to be a primer on SSH server and client security. There are too many features to discuss, and we must address greater issues than just technical settings.)

So what can your organization do to help secure multiple versions of SSH running on multiple operating systems?

Mitigating Organizational Risk

Why would anyone propose running SSH on most major platforms? For more than a decade, architectures have settled on standard utilities and protocols. For example, Telnet and FTP are commonly used on all kinds of servers, including some established long before TCP/IP, UNIX, the C programming language, etc. But while Telnet and FTP are common denominators, they’re lowest common denominators when you consider their security issues. In response, organizations look to SSH to take the place of these insecure protocols. SSH is very scriptable and makes automation easy by allowing common code to run on several platforms; for example, by using Perl. Instead of working the script through two separate utilities, you can stay "in session" with SSH. File size seem too small? Kick off another transfer via SSH seamless processing power that allows both file transfer and command execution. It even allows for authentication to work without pesky passwords and authentication processes. It encrypts all traffic, so the security people feel warm and fuzzy. It’s an easy sell.

Now, how do you make sure that all those administrators and platform security architectures are on the same page? You must create a cross-platform SSH implementation team that will take ownership of the issues. This group can ensure that security is consistent across platforms, by identifying implementation weaknesses on some of those platforms.

For example, you must control access to the private-key file used with authentication. Imagine that the desktop Windows administrator has no clue how important PCs are to server security. He or she has implemented the FAT file system, making any sort of file access security very difficult. To make remote administration easy, the same person has opened the administrative shares and Remote Desktop access to Everyone. With this kind of design, a hacker can collect private keys like flowers for the plucking. With these private keys (used with multiplatform scripts), the hacker can take on any identity s/he wants—the time needed varying only in relationship to the private key’s passphrase complexity.

Don’t want me to pick on Windows administrators? Perhaps you’re serving out home directories with NFS instead. Have you studied NFS security? Got poor home directory permissions on top of poor NFS permissions?

This is just one example of how one platform’s foolishness endangers all others. Once I heard an administrator strongly recommend that we all use the same host key on our servers. This approach would remove the pesky warning that users get regarding host identity. How likely is phishing in a big intranet? Here again, one person’s ideas remove a vital security component from SSH. The best way to have a consistent, complete security model is to have a group of experts assemble the plan, document it, and weave it into server and client implementation and maintenance processes.

This group must be empowered to choose the best settings, despite complaints from administrators and developers alike. The group must be responsible for security openings found during an intrusion study performed by a credible external group. You must confirm the security plan, right? The group should find a monitoring system, either purchased or self-written, that checks production settings. You might even create a process that periodically overwrites existing files with a baseline file containing documented security settings. (It’s amazing what vendors, customers, etc. with root access can do to your settings files!)

Let me warn you: SSH security begins with use of the server-side security features. There must be agreement on which server settings are optional and which are mandatory, with which specific settings enabled. Is SSH protocol 1 good enough? One version of SSH has a configuration file with security features commented out. It’s assumed that you—and the hacker 0wning your server—have compiled SSH with security embedded. Is that good enough, and did anyone on your team catch that fail-open security design from the factory?

There will be some tough, very detailed questions to answer. If reading books and articles isn’t enough, management should work with a consultant whose costs are shared across all platform owners.

Mitigating Agent-Forwarding Risk

I wish I could describe an elegant security design for agent forwarding—I like it that much. But devices in untrusted zones must not be trusted. No matter how much people complain, SSH services running in untrusted zones must have agent forwarding disabled. If agent forwarding is enabled, the implementation will allow an intruder to connect to the stored authentication credentials and then use them to connect to other DMZ hosts (or even intranet hosts, if port forwarding has been allowed as well). Disable agent forwarding by default and allow it as needed. Don’t assume that the risk isn’t there if your platform’s software and online help don’t mention it. Specifically, disable it in the server’s configuration. It may be put in as a compatibility feature, just waiting for someone to activate it.

What about using an SSH feature that allows SSH to send a notification to your terminal when authentication credentials are used, via the X Window System? Of course, this option is limited to one distribution at this time. I’m not even sure that it would work to stop an intruder from using your credentials. But maybe this is the control to agent forwarding? Review the ssh-add man page. Meanwhile, let’s discuss the X Window System (we’ll refer to it as X11) and DMZ hosts.

X11 has many good points, but its security record has issues. Most people don’t understand the inverted relationship. As you use X11, the remote device writes to your X11 server. This server runs as root in UNIX and can run processes as root on your device (if the code needs updates). Many people disable access control to get X11 to work, and a few X11 emulators that run on Windows install with access control disabled. This technique allows hackers to plant loggers to your display and record keystrokes or make screen captures. Additionally, they can force pop-ups on you that harvest information such as SSH passphrases. I wouldn’t give a DMZ host X11 access to an internal device, especially if it’s done to enable a possible SSH security feature that exists with one SSH platform only.

Agent forwarding is a useful feature, but most vendors recommend against enabling it in all environments. Are intranets that trustworthy, given all the malicious insider activity? At some point, the mitigation shifts from banning/allowing agent forwarding right into detecting misuse of agent forwarding. For now, create a ban on the feature outside of the intranet. From there, you simply need a robust security architecture that can detect and respond to odd access events.

Mitigating Port-Forwarding Risk

Port forwarding is risky because many people don’t understand how it works. Many system administrators skip over the details in a mad rush to create an FTP replacement system. Additionally, many admins are used to client software being relatively harmless. They don’t understand how many features are available within most SSH clients. So those tunneling abilities remain a threat—a hidden, ignored threat.

How much open, unfettered access does your organization have to devices on the Internet?

SSH isn’t the only service that can do port forwarding. There are standalone port forwarders, such as netcat (nc), that can complete the hack. But do you expect something you see as a Telnet replacement to be its own VPN tool?

How do you mitigate the risk? After all, you can’t control the Internet servers, and most SSH clients have port forwarding capabilities. Even if your "official" client doesn’t, the malicious user can install another. And though you may have your firewall blocking port 22 outbound, that’s not good enough. SSH can run on any TCP port. So the home SSH server runs on the email port, the DNS port, or any port that people commonly open for outbound access. This seems an unsolvable riddle.

Let’s begin with the basics. People can’t comply with unwritten policy. You must create and communicate, strongly, a policy against creating tunnels/remote access into your organization. Emphasize that any tunneling/encryption technologies that go outside of your official remote access program and VPN toolsets are forbidden.

Do you even have a remote access and/or VPN strategy? Or do you allow just about anyone to use anything handy as a conduit to your precious business secrets? You must state the consequences for creating a private VPN into YourOrg.com—even if it seems unlikely that you can detect these tunnels.

Allowing people to run personal computers with administrative authority allows them to evade any security control or secure software alternative you provide. Once you have more control on the SSH client software being run, create a process to survey and alert based on client settings (or simply overwrite them with the correct settings). While you can’t control a remote server’s configuration, you can control the client-side settings running on your organization’s PCs. Look hard—you may find a PC client that works with Active Directory and group policies.

Now that people are warned and the PCs are set such that tunnels don’t happen by accident, it’s time to tighten default accesses. The idea is that any tunnel, if found, is plainly a malicious act that involves knowingly and consciously configuring to evade policy and to deceive the organization.

Create a review process for approving SSH access to key devices on the Internet. Sure, SSH can be run on multiple ports or even tunneled into something like HTTP. But you must set the precedent that by default outbound SSH is not allowed. Consider high-level inspection tools that can detect tunneled traffic. Some proxy servers will inspect HTTP traffic for compliance to HTTP packet standards. If the HTTP packet is bad, the packet is dropped. But what about HTTPS tunneling? Can that be inspected?

Inspect firewall and WAN router logs for long-term, persistent connections or for those that begin at odd times and go outside your network. If the destination is outside of your approval process, you may have a stealth tunnel in operation. Time to ask a few questions or do some monitoring of the scripts running on the PC. While PC implies "personal," they are company properties and therefore open to monitoring.

You might even break the tunnel to help monitor it. Simply state that all jobs that access external resources must initiate on a DMZ host that you control. This application proxying is a valuable way to control outbound access. Proxying allows you to monitor information flows and prevent port forwarding from burrowing into your intranet itself.

Final Ideas

As you can see, SSH security is not easy to implement. Before your organization implements SSH, all target platforms must be in synch on the risks and the mitigations each will implement. One weak platform will unravel all your security quickly. Your security staff must consider risks beyond technical settings; therefore, you must do the same.

Implementing SSH security requires at least a book, if not some hired help. As a security tool, it includes sophisticated VPN, authentication, and process execution and control abilities. This ain’t no mere FTP drop-in replacement.

Remember, using agent forwarding in untrusted environments is risky, and allowing an open interface to the Internet allows unmonitorable tunnels into and out of your company. It’s a big problem, requiring a response from experts. Remediation is not easy; at some point you must inspect the application layer—difficult with encrypted traffic. The best answers are a team approach to the rollout and possibly bringing in the right supplements to your rollout. If you get it right—wow, what a tool! Get it wrong, however, and you’ll have deep weaknesses on your platforms using SSH, wherever they are in your network.