Securing Sun Cluster 3.x Nodes
Building a secure system requires that entry points into the system be limited and restricted, in addition to limiting how authorized users obtain privileges. To effectively secure each node in a cluster, you must make changes to the Solaris OE software running on each node.
Properly securing the Sun Cluster 3.x nodes requires the following:
- "Installing Solaris OE and Sun Cluster Software" on page 12
- "Adding Security Software" on page 12
- "Installing Downloaded Software and Implementing Modifications" on page 18
- "Verifying Node Hardening" on page 20
- "Reviewing the Results" on page 21
Installing Solaris OE and Sun Cluster Software
At this point in the process, we assume that you have installed the appropriate Solaris OE, Solaris OE cluster on the cluster nodes, and the Sun Cluster 3.x software. Also, we assume that you have configured the software.
Install the security software only if the cluster is installed and running with the appropriate agents. (Refer to "Supported Agents" on page 4.)
Adding Security Software
The next stage in hardening Sun Cluster 3.x nodes requires downloading and installing additional software security packages. This section covers the following tasks:
- "To Install Solaris Security Toolkit Software" on page 13
- "To Download Recommended and Security Patch Software" on page 14
- "To Download FixModes Software" on page 15
- "To Download OpenSSH Software" on page 16
- "To Download MD5 Software" on page 17
Of the software described in this section, the Solaris Security Toolkit, Recommended and Security Patch Cluster, FixModes, and MD5 software are required. The use of a Secure Shell product is strongly recommended and is bundled with Solaris 9 OE. For Solaris 8 OE the use of OpenSSH is recommended, though not required. Instead of OpenSSH, you can substitute a commercial version of Secure Shell, available from a variety of vendors.
To Install Solaris Security Toolkit Software
The Solaris Security Toolkit version 0.3.10 software must be downloaded first, then installed on each of the nodes. Later, you'll use the Solaris Security Toolkit to automate installing other security software and implementing the Solaris OE modifications for hardening the system.
The primary function of the Solaris Security Toolkit software is to automate and simplify building secured Solaris OE systems based on the recommendations contained in this and other security-related Sun BluePrints OnLine articles.
The following instructions use file names that are correct only for version 0.3.10 of the Solaris Security Toolkit software.
Download the source file (SUNWjass-0.3.10.pkg.Z) from the following web site:
Extract the source file into a directory on the server using the uncompress command:
Install the Solaris Security Toolkit software on the server using the pkgadd command:
# uncompress SUNWjass-0.3.10.pkg.Z
# pkgadd -d SUNWjass-0.3.10.pkg SUNWjass
Executing this command creates the SUNWjass subdirectory in /opt. This subdirectory contains all Solaris Security Toolkit software directories and associated files. The script make-jass-pkg, included in Solaris Security Toolkit software releases since version 0.3, allows administrators to create custom packages using a different installation directory.
To Download Recommended and Security Patch Software
Patches are regularly released by Sun to provide Solaris OE fixes for performance, stability, functionality, and security. It is critical to the security of a system that the most up-to-date patch is installed. To ensure that the latest Solaris OE Recommended and Security Patch is installed, this section describes how to download the latest patch cluster and store them uncompressed in the /opt/SUNWjass/Patches directory on each node.
Downloading the latest patch cluster does not require a SunSolveSM program support contract.
Apply standard best practices to all patch installations. Before installing any patches, evaluate and test them on non-production systems or during scheduled maintenance windows.
Download the latest patch from the SunSolve OnlineSM web site at:
Click on the "Patches" link, at the top of the left navigation bar.
Select the appropriate Solaris OE version in the Recommended Solaris Patch Clusters box.
Select the best download option, either HTTP or FTP, with the associated radio button, then click Go.
Save the file locally.
Move the file securely to the node being hardened using the scp SSH command, the sftp SSH command, or the ftp command (if SSH is not yet installed).
Move the file to the /opt/SUNWjass/Patches directory and uncompress it.
In our example, we select Solaris 8 OE.
A Save As dialog box is displayed in your browser window.
The scp command used to copy the file to a domain called scnode01 should appear similar to the following:
% scp 8_Recommended.zip scnode01:/var/tmp
The following commands perform these tasks:
# cd /opt/SUNWjass/Patches # mv /var/tmp/8_Recommended.zip . # unzip 8_Recommended.zip Archive: 8_Recommended.zip creating: 8_Recommended/ inflating: 8_Recommended/CLUSTER_README inflating: 8_Recommended/copyright inflating: 8_Recommended/install_cluster [. . .]
Later, using the Solaris Security Toolkit software, you'll install the patch after downloading all the other security packages.
If you do not place the Recommended and Security Patches software into the /opt/SUNWjass/Patches directory, a warning message displays when you execute the Solaris Security Toolkit software.
To Download FixModes Software
FixModes is a software package that tightens the default Solaris OE directory and file permissions. Tightening these permissions can significantly improve overall security. More restrictive permissions make it even more difficult for malicious users to gain privileges on a system.
Download the FixModes precompiled binaries from:
Save the file, FixModes.tar.Z, in the Solaris Security Toolkit software Packages directory in /opt/SUNWjass/Packages.
The FixModes software is distributed as a precompiled and compressed tar file formatted for SPARC_ based systems. The file name is FixModes.tar.Z.
Leave the file in its compressed state.
Later, using the Solaris Security Toolkit software, you'll install the FixModes software after downloading all the other security packages.
To Download OpenSSH Software
In any secured environment, the use of encryption in combination with strong authentication is required to protect user-interactive sessions. At a minimum, user interactive sessions must be encrypted.
The tool most commonly used to implement encryption is SSH software, whether a commercial or open source (freeware) version. To implement all the security modifications performed by the Solaris Security Toolkit software and recommended in this article, you need to implement a SSH software product.
The Solaris Security Toolkit software disables all non-encrypted user-interactive services and daemons on the system, in particular daemons such as in.rshd, in.telnetd, and in.ftpd. Access to the system can be gained with SSH similarly to what is provided by RSH, Telnet, and FTP.
The Solaris 9 OE includes a bundled version of OpenSSH called the Solaris Secure Shell. The use of this software is highly recommended on Solaris 9 OE Sun Cluster environments because Solaris Secure Shell is supported by Sun. If you are using this product, then it is not necessary to perform the steps in this section. If you are not using Solaris Secure Shell on Solaris 9 OE, then use an alternative SSH version so that you can implement all of the recommended security modifications.
If you choose to use an SSH product other than OpenSSH or the bundled Solaris Secure Shell, install and configure it before or during a Solaris Security Toolkit run.
Obtain the following online article and use the instructions in that article for downloading the software.
A Sun BluePrints OnLine article about how to compile and deploy OpenSSH titled "Building and Deploying OpenSSH on the Solaris Operating Environment" is available at:
Later, using the Solaris Security Toolkit software, you'll install the OpenSSH software after downloading all the other security packages.
If possible, do not compile OpenSSH on the cluster nodes and do not install the compilers on the cluster nodes. Use a separate Solaris OE systemrunning the same Solaris OE version, architecture, and mode (for example, Solaris 8 OE, Sun4u, and 64-bit)to compile OpenSSH. If you implement a commercial version of SSH, then no compilation is required. The goal is to limit the availability
To Download MD5 Software
The MD5 software validates MD5 digital fingerprints on the cluster. Validating the integrity of Solaris OE binaries provides a robust mechanism to detect system binaries that are altered or trojaned (hidden inside something that appears safe) by unauthorized users. By modifying system binaries, attackers provide themselves with backdoor access onto a system; they hide their presence and cause systems to operate in unstable manners.
To install the MD5 program (Intel and SPARC architectures), follow these steps:
Download the MD5 binaries from:
Save the downloaded file, md5.tar.Z, to the Solaris Security Toolkit Packages directory in:
(Optional) Download and install Solaris Fingerprint Database Companion and Solaris Fingerprint Database Sidekick software from the SunSolve Online web site at:
The MD5 programs are distributed as a compressed tar file.
Do not uncompress the tar archive.
After the MD5 software has been saved to the /opt/SUNWjass/Packages directory, it is installed during the execution of the Solaris Security Toolkit software.
After the MD5 binaries are installed, you can use them to verify the integrity of executables on the system through the Solaris Fingerprint Database. More information on the Solaris Fingerprint Database is available in the Sun BluePrints OnLine article titled "The Solaris_ Fingerprint Database - A Security Tool for Solaris Software and Files:"
We strongly recommend that you install these optional tools and use them with the MD5 software. These tools simplify the process of validating system binaries against the database of MD5 checksums. Use these tools frequently to validate the integrity of the Solaris OE binaries and files on the cluster nodes.
These tools are described in the "The Solaris_ Fingerprint Database - A Security Tool for Solaris Software and Files" Sun BluePrints Online article.
Installing Downloaded Software and Implementing Modifications
Now that all software is downloaded and added to the correct directories, each of the Solaris OE images running on each of the Sun Cluster 3.x nodes can be secured.
The Solaris Security Toolkit version 0.3.10 provides a driver (suncluster3x-secure.driver) for automating the installation of security software and Solaris OE modifications. The driver for the cluster nodes performs the following tasks:
Installs and executes the FixModes software to tighten file system permissions
Installs the MD5 software
Installs the Recommended and Security Patch software
Implements over 100 Solaris OE security modifications
During the installation and modifications implemented in this section, all non-encrypted access mechanisms to the nodessuch as Telnet, RSH, and FTPare disabled. The hardening steps do not disable console serial access over serial ports.
Harden only one cluster node at a time. Before hardening the next node, verify that the hardened configuration functions properly in your environment.
To Install Downloaded Software and Implement Modifications
Disable the failover mode before hardening a node.
On one node only, execute the suncluster3x-secure.driver script as follows:
Verify that the node is hardened.
Re-enable failover only after each node is hardened, rebooted, and tested.
After you complete the hardening process for a node, use the procedure in "Verifying Node Hardening" on page 20 to verify that the configuration.
# cd /opt/SUNWjass # ./jass-execute -d suncluster3x-secure.driver ./jass-execute: NOTICE: Executing driver, suncluster3x-secure.driver ============================================================ suncluster3x-secure.driver: Driver started. ============================================================ [...]
By executing the suncluster3x-secure.driver script, all of the security modifications included in the script are made on the system. The current release of this driver includes over 100 security modifications to the Solaris OE image running on each node of the cluster.
The suncluster3x-secure.driver automatically executes the FixModes program, which must be already downloaded and filed in the correct directory as described previously, to tighten file system permissions on the system.
When the Solaris Security Toolkit program executes the FixModes program, the following warning messages are displayed:
Installing 'fix-modes' into ///opt/FixModes Executing 'fix-modes' from ///opt/FixModes. secure-modes: WARNING: Can't find required uid/gid smmsp secure-modes: WARNING: Can't find required uid/gid smmsp
These are known warning messages from FixModes that can be safely ignored.
To view the contents of the driver file and obtain information about the Solaris OE modifications, refer to the Solaris Security Toolkit documentation available either in the /opt/SUNWjass/Documentation directory or through the web at:
For information about other scripts in the Solaris Security Toolkit software, refer to the Solaris Security Toolkit documentation available from this URL.
Refer to "Verifying Node Hardening" on page 20 for detailed instructions.
This task keeps the cluster software from failing over to a hardened node before it is fully hardened and before the hardened configuration is validated.
Verifying Node Hardening
After performing the procedures in this article to harden a cluster node, test the configuration and hardening.
The number of daemons and services running on each of the nodes is significantly less after the hardening steps are completed. For the example configuration, our testing resulted in the following:
On the node where these recommendations were tested, the number of Solaris TCP services listed by netstat decreased from 31 to 7.
The number of UDP IPv4 services listed by netstat went from 57 to 6.
By reducing the number of services available, the exposure points of this system are significantly reduced and the security of the entire cluster is dramatically improved.
We recommend that you disable the failover before hardening any of the nodes. Re-enable failover only after each node has been hardened, rebooted, and tested. This practice avoids having the cluster software fail over to a hardened node before it has been fully hardened and before the hardened configuration has been validated.
To Verify Node Hardening
After you complete the hardening process for a node, reboot the node and verify its configuration by having it assume the appropriate Sun Cluster 3.x software role.
When the hardened node takes control of the cluster, verify the node's functionality.
After verifying that the node is functioning properly, perform the entire software installation and hardening process on the next node.
This step must be done before you harden any other nodes in the cluster.
Do not harden other Sun Cluster nodes before verifying that the hardened configuration of each node functions properly in your environment.
Refer to "Securing Sun Cluster 3.x Nodes" on page 12 for the procedure.
Do not harden all nodes simultaneously.
Reviewing the Results
After you harden a node, you can review the results of the Solaris Security Toolkit run, to determine if fewer services are enabled.
To Review the Results of a Run
Review the run directory in /var/opt/SUNWjass/run.
Each Solaris Security Toolkit software run creates a run directory in /var/opt/SUNWjass/run. The names of these directories are based on the date and time the run is initiated. In addition to displaying the output to the console, the Solaris Security Toolkit software creates a log file in the /var/opt/SUNWjass/run directory.
Do not modify the contents of the /var/opt/SUNWjass/run directories under any circumstances. Modifying the files can corrupt the contents and cause unexpected errors when you use Solaris Security Toolkit software features such as undo.
The files stored in the /var/opt/SUNWjass/run directory track modifications performed on the system and enable the jass-execute undo feature. You can undo a run, or series of runs, with the jass-execute -u command. For example, on a system where two separate Solaris Security Toolkit runs are performed, you could undo them by using the following command:
# pwd /opt/SUNWjass # ./jass-execute -u Please select from one of these backups to restore to 1. September 25, 2001 at 06:28:12 (/var/opt/SUNWjass/run/20010925062812) 2. April 10, 2001 at 19:04:36 (/var/opt/SUNWjass/run/20010410190436) 3. Restore from all of them Choice? 3 ./jass-execute: NOTICE: Restoring to previous run //var/opt/SUNWjass/run/20010410190436 ============================================================ undo.driver: Driver started. ============================================================ [...]
Refer to the Solaris Security Toolkit documentation for details on the capabilities and options available in the jass-execute command.
Based on our qualified configuration, we find the following noncluster services remain running on a hardened node:
# ps -ef | grep -v cluster UID PID PPID C STIME TTY TIME CMD root 0 0 0 Oct 25 ? 0:01 sched root 1 0 0 Oct 25 ? 0:00 /etc/init - root 2 0 0 Oct 25 ? 0:00 pageout root 3 0 0 Oct 25 ? 4:41 fsflush root 466 1 0 Oct 25 ? 0:00 /usr/lib/saf/sac -t 300 root 65 1 0 Oct 25 ? 0:01 /usr/lib/sysevent/syseventd root 67 1 0 Oct 25 ? 0:00 /usr/lib/sysevent/syseventconfd root 77 1 0 Oct 25 ? 8:22 devfsadmd root 265 1 0 Oct 25 ? 0:00 /usr/lib/netsvc/yp/ypbind / -broadcast root 252 0 Oct 25 ? 0:00 /usr/sbin/rpcbind root 167 1 0 Oct 25 ? 0:00 /usr/sbin/in.rdisc -s root 469 466 0 Oct 25 ? 0:00 /usr/lib/saf/ttymon root 255 1 0 Oct 25 ? 0:00 /usr/sbin/keyserv -d root 394 1 0 Oct 25 ? 0:00 /usr/lib/utmpd root 274 1 0 Oct 25 ? 0:00 /usr/sbin/inetd -s -t root 318 1 0 Oct 25 ? 0:00 /usr/lib/inet/xntpd root 285 1 0 Oct 25 ? 0:00 /usr/sbin/syslogd -t root 327 274 0 Oct 25 ? 0:00 rpc.metad root 396 1 0 Oct 25 ? 0:00 /usr/sbin/nscd root 373 1 0 Oct 25 ? 0:00 /usr/sbin/cron root 391 1 0 Oct 25 ? 0:00 /usr/sbin/vold root 470 1 0 Oct 25 ? 0:00 /usr/lib/sendmail -q15m root 1060 1 0 13:54:45 ? 0:00 /opt/OBSDssh/sbin/prngd --cmdfile /etc/prngd.conf -seedfile /var/spool/prngd/p root 1086 1 1 13:55:00 ? 0:00 /opt/OBSDssh/sbin/sshd
The preceding listing of services may not exactly match your environment. We made several configuration modifications on this node after the operating system was installed. These modifications include the configuration of xntp and NIS, and the installation of OpenSSH.
The following output is generated for our sample configuration by Nmap, a popular freeware security scanning tool:
# nmap -p 1-65535 10.6.25.150 Starting nmap V. 2.53 by firstname.lastname@example.org (http://www.insecure.org/nmap/ ) Port State Service 22/tcp open ssh 111/tcp open sunrpc 8059/tcp open unknown 8060/tcp open unknown 32785/tcp open unknown 32786/tcp open sometimes-rpc25 32787/tcp open sometimes-rpc27 32788/tcp open unknown 32789/tcp open unknown 32790/tcp open unknown 32791/tcp open unknown 32804/tcp open unknown 32806/tcp open unknown 32811/tcp open unknown 32821/tcp open unknown Nmap run completed -- 1 IP address (1 host up) scanned in 211 seconds
Ports 8059 and 8060 are Sun Cluster 3.x software-specific ports that accept connections only from other cluster nodes. When a connection request from a noncluster node is received, the following message is logged to syslog:
Oct 30 14:00:52 phys-sps-1 cl_runtime: WARNING: Received a connect request from a node not configured in the cluster. Nodeid 0 ipaddr 0x8194b556
Monitor log files for these types of messages so that appropriate action can be taken when unauthorized access attempts are made against a cluster.
Cluster nodes are added based on the authentication method defined in the Sun Cluster 3.x software configuration. We recommend that you use the strongest possible method of authentication. The available options are described in "Node Authentication Options" on page 10.