Securing the Sun Fire 12K/15K System Controller
This article recommends how to securely deploy system controllers (SCs) on Sun Fire TM 12K and 15K systems. These recommendations apply to environments where security is a concern, particularly environments where the uptime requirements of the SCs and/or the information on the Sun Fire server is critical to the organization. This article was updated for System Management Services (SMS) 1.4 software.
The SCs control the hardware components that comprise a Sun Fire 12K or 15K server. Because they are a central control point for the entire frame, they represent an attack point for intruders. To improve reliability, availability, serviceability, and security (RASS), the SCs must be secured against malicious misuse and attack.
This article is one in a series that provides recommendations for enhancing the security of a Sun Fire system. After securing the SCs, we recommend that you use the "Securing the Sun Fire 12K and 15K Domains" article to secure the Sun Fire domains.
This article contains the following topics:
"Securing the System Controller"
"Verifying SC Hardening"
"About the Authors"
The following sections provide helpful information for understanding the SCs, hardware and software requirements, and other topics. This section contains the following topics:
"Assumptions and Limitations"
"Understanding the System Controller"
"Default SMS Configuration"
"SC Network Interfaces"
"Security Options in SMS 1.4"
Assumptions and Limitations
In this article, our recommendations are based on several assumptions and limitations as to what can be done to secure Sun Fire SCs, resulting in a supported configuration.
Our recommendations are based on Solaris TM 8 Operating Environment (2/02) or Solaris 9 Operating Environment running System Management Services software version 1.4 on the SCs. All of the Solaris Operating Environment (Solaris OE) components described in this article are included in these releases.
The recommendations in this article are for System Management Services (SMS) 1.4 software, and differences between SMS 1.4 and previous releases are not discussed. It is recommended that all customers upgrade their software to SMS 1.4 when possible.
Solaris Operating Environment (Solaris OE) hardening can be interpreted in many ways. For purposes of developing a hardened SC configuration, we address hardening all possible Solaris OE options. That is, anything that can be hardened is hardened. When there are good reasons for leaving services and daemons as they are, we do not harden or modify them.
Be aware that hardening Solaris OE configurations to the level described in this article may not be appropriate for your environment. For some environments, you may want to perform fewer hardening operations than recommended here. The configuration remains supported in these cases; however, additional hardening beyond what is recommended in this article is not supported.
You can customize a copy of the Sun Fire 12K and 15K SC module of the Solaris Security Toolkit to disable certain hardening scripts. It is strongly recommended that any modifications to the default modules be made in copies of those files to simplify upgrades to newer Solaris Security Toolkit versions.
Solaris OE minimization (removing Solaris OE packages to minimize security exposures) is not a generally supported option on the SCs. The baseline supported OE configuration for Sun Fire SCs requires the SUNWCall Solaris OE cluster. However, it is possible to create a supported exception to this rule. If you are interested in minimizing the SCs, contact your Sun account team for assistance.
Standard security rules apply to the hardening of SCs: That which is not specifically permitted is denied.
In this article, we omit additional software that you can install on the SCs, such as SunSM Remote Services Event Monitoring, SunSM Remote Services Net Connect, and Sun TM Management Center software. We recommend that you carefully consider the security implications implicit with the installation of these types of software.
The SC configuration for Sun Fire systems implemented by the Solaris TM Security Toolkit module (sunfire_15k_sc-secure.driver) is a Sun supported configuration. A hardened SC is supported only if the security modifications are performed using the Solaris Security Toolkit. Support calls to Sun's support services are handled the same as other cases.
The Solaris Security Toolkit itself is not a supported Sun product. Only configurations created with the Solaris Security Toolkit are supported.
To obtain Solaris Security Toolkit support, use the Solaris Security Forum link at the following web site:
Understanding the System Controller
Securing the SCs is the first priority in configuring Sun Fire systems to be resistant to unauthorized access and to function properly in hostile environments. Before securing the SCs, it's important to understand the services and daemons that are running on the system. This section describes the software, services, and daemons specific to the SCs. The functionality is described at a high-level with references to Sun documentation for more detailed information. This section provides administrators with a baseline of functionality required for the SCs to perform properly.
The SCs are multifunction system boards within the Sun Fire frame. This system is dedicated to running the SMS software. The SMS software is used to configure dynamic domains, provide console access to each domain, control whether a domain is powered on or off, and provide other functions critical to operating and monitoring Sun Fire systems.
The following list is an overview of the many services the SCs provide for the Sun Fire systems:
Manages the overall system configuration.
Acts as a boot initiator for its domains.
Serves as the syslog host for its domains; note that an SC can still be a syslog client of a LAN-wide syslog host.
Provides a synchronized hardware clock source.
Sets up and configures dynamic domains.
Monitors system environmental information, such as power supply, fan, and temperature status.
Hosts field-replaceable unit (FRU) logging data.
Provides redundancy and automated SC failover.
Provides a default name service for the domains based on virtual hostids, and MAC addresses for the domains.
Provides administrative roles for frame management.
Sun Fire frames have two SCs. Our security recommendations are the same for both SCs. The SC that controls the platform is referred to as the main SC, while the other SC acts as a backup and is called the spare SC. The software running on the SC monitors the SCs to determine when an automatic failover should be performed.
For our sample configuration, the main SC is sc0 and the spare SC is sc1.
We recommend that the two SCs have the same configuration. This duplication includes the Solaris OE, security modifications, patch installations, and all other system configurations.
The failover functionality between the SCs is controlled by the daemons running on the main and spare SCs. These daemons communicate across private communication paths built into the Sun Fire frames. Other than the communication of these daemons, there is no special trust relationship between the two SCs.
A significant aspect of SC security is access to applications that an administrator uses to manage Sun Fire systems. Some security issues associated with the SMS software are described in the System Management Services (SMS) 1.4 Administrative Guide. In this article, we build on the recommendations made in that guide.
Access to the SMS software on the SCs is a core concern. Correspondingly, access to this software must be carefully controlled. Only authorized users should have access. The SMS software provides a mechanism, over and above the Solaris OE access controls, to limit access to the SMS software. These features are described in "Default SMS Configuration" on page 6.
Default SMS Configuration
This section describes the default SMS configuration installed on Sun Fire SCs.
Sun Fire SCs running SMS 1.4 are based on Solaris 8 OE (2/02) or Solaris 9 OE, using the SUNWCall Solaris OE installation cluster.
The SMS software resides on the SCs and oversees all SC operations. The entire SMS software bundle is comprised of the following packages, which are specific to the Sun Fire 12K and 15K SCs:
application SUNWSMSdf System Management Services Data Files application SUNWSMSjh System Management Services On-Line Javahelp application SUNWSMSlp System Management Services LPOST object files application SUNWSMSmn System Management Services On-Line Manual Pages application SUNWSMSob System Management Services OpenBoot PROM application SUNWSMSod System Controller Open Boot Prom application SUNWSMSop System Management Services Core Utilities application SUNWSMSpd System Controller Power On Self Test application SUNWSMSpo System Management Services POST Utilities application SUNWSMSpp System Management Services picld(1M) Plug-in Module application SUNWSMSr System Management Services, (Root) application SUNWSMSsu System Management Services User Environment application SUNWufu User Flash PROM Device Driver Header File application SUNWufrx User Flash PROM Device Driver (Root) (64-bit) application SUNWscdvr Sun Fire 15000 System Controller drivers
Accounts and Security
The following users are added to the /etc/passwd file by the SMS software:
# grep sms /etc/passwd sms-codd:x:10:2:SMS Capacity On Demand Daemon:: sms-dca:x:11:2:SMS Domain Configuration Agent:: sms-dsmd:x:12:2:SMS Domain Status Monitoring Daemon:: sms-dxs:x:13:2:SMS Domain Server:: sms-efe:x:14:2:SMS Event Front-End Daemon:: sms-esmd:x:15:2:SMS Environ. Status Monitoring Daemon:: sms-fomd:x:16:2:SMS Failover Management Daemon:: sms-frad:x:17:2:SMS FRU Access Daemon:: sms-osd:x:18:2:SMS OBP Service Daemon:: sms-pcd:x:19:2:SMS Platform Config. Database Daemon:: sms-tmd:x:20:2:SMS Task Management Daemon:: sms-svc:x:6:10:SMS Service User:/export/home/sms-svc:/bin/csh
Of these accounts, sms-svc is the only default account that administers the system. All the other accounts provide privileges for the daemons they are associated with. Never use these accounts to log into the system. You can secure them the same way as unused system accounts. These accounts are for the daemons running the SCs as described in "SMS Daemons" on page 9.
The following are newly added SMS /etc/shadow contents:
# grep sms /etc/shadow sms-codd:NP::::::: sms-dca:NP::::::: sms-dsmd:NP::::::: sms-dxs:NP::::::: sms-efe:NP::::::: sms-esmd:NP::::::: sms-fomd:NP::::::: sms-frad:NP::::::: sms-osd:NP::::::: sms-pcd:NP::::::: sms-tmd:NP::::::: sms-svc:lnrf21Ovf4G9s:11414::::::
All of these accounts, including the sms-svc account, are initially locked with "NP" as the encrypted password entry.
Set the password for the sms-svc user on both SCs immediately after installing the SMS software or first powering on the system.
The following entries are added to the /etc/group file by the SMS software:
# grep sms /etc/group platadmn::15:sms-svc platoper::16:sms-svc platsvc ::17:sms-svc dmnaadmn::18:sms-svc dmnarcfg::19:sms-svc dmnbadmn::20:sms-svc dmnbrcfg::21:sms-svc dmncadmn::22:sms-svc dmncrcfg::23:sms-svc dmndadmn::24:sms-svc dmndrcfg::25:sms-svc dmneadmn::26:sms-svc dmnercfg::27:sms-svc dmnfadmn::28:sms-svc dmnfrcfg::29:sms-svc dmngadmn::30:sms-svc dmngrcfg::31:sms-svc dmnhadmn::32:sms-svc dmnhrcfg::33:sms-svc dmniadmn::34:sms-svc dmnircfg::35:sms-svc dmnjadmn::36:sms-svc dmnjrcfg::37:sms-svc dmnkadmn::38:sms-svc dmnkrcfg::39:sms-svc dmnladmn::40:sms-svc dmnlrcfg::41:sms-svc dmnmadmn::42:sms-svc dmnmrcfg::43:sms-svc dmnnadmn::44:sms-svc dmnnrcfg::45:sms-svc dmnoadmn::46:sms-svc dmnorcfg::47:sms-svc dmnpadmn::48:sms-svc dmnprcfg::49:sms-svc dmnqadmn::50:sms-svc dmnqrcfg::51:sms-svc dmnradmn::52:sms-svc dmnrrcfg::53:sms-svc
Groups provide the groundwork for delegation of domain and chassis administrative capabilities. They allow for separation of the administrative privileges and operator privileges for each domain and the entire frame. The System Management Services (SMS) 1.4 Administrator Guide contains detailed descriptions of commands that require a group's privileges for executing.
The SMS daemons are divided into the following three types, with sample ps output.
First are the platform or core SMS daemons run on both the main and spare SC:
root 8108 1 0 17:53:04 ? 0:01 mld root 8123 1 0 17:53:05 ? 31:35 hwad root 8126 1 0 17:53:05 ? 0:00 mand sms-frad 331 1 0 12:41:21 ? 0:00 frad root 8132 1 0 17:53:06 ? 0:03 fomd root 4830 1 0 09:35:56 ? 0:00 ssd -i SMS software start-up initiated -iSC POST results: 'CP1500 POST Passed;'
Next are the SMS daemons that run only on the main SC:
sms-pcd 393 1 0 12:41:43 ? 0:03 pcd sms-tmd 402 1 0 12:41:43 ? 0:00 tmd -t 12 sms-dsmd 405 1 0 12:41:44 ? 0:00 dsmd sms-esmd 414 1 0 12:41:45 ? 0:05 esmd sms-osd 419 1 0 12:41:46 ? 0:00 osd root 8218 1 0 17:53:33 ? 0:00 kmd sms-efe 475 1 0 12:41:47 ? 0:00 efe sms-codd 483 1 0 12:41:48 ? 0:00 codd
Third are the SMS daemons that communicate to the domains, which run only on the main SC:
sms-dxs 4428 291 0 13:14:31 ? 0:00 dxs -d A sms-dca 4429 291 0 13:14:31 ? 0:00 dca -d A
The previous list of domain services is a sample of the services that may be encountered. Depending on how many domains are in use, more SMS daemons may be running.
These SMS daemons are started by /etc/rc3.d/S99sms.
The SMS startup daemon was moved from /etc/rc2.d to /etc/rc3.d with the release of SMS 1.3
The following paragraphs briefly describe the SMS daemons. For additional information on each of these daemons, refer to the System Management Services (SMS) 1.4 Administrator Guide and System Management Services (SMS) 1.4 Reference Guide.
This daemon (domain configuration administration) supports remote dynamic reconfiguration (DR) by facilitating communication between applications and the dca daemon running on the domain. A separate instantiation of the dca daemon is run on the main SC for each domain running Solaris OE.
This daemon (domain status monitoring daemon) monitors domain state, CPU reset conditions, and the Solaris OE heartbeat for all domains. This daemon notifies the dxs daemon and Sun Management Center software of all changes.
This daemon (domain x server) provides a variety of software support for a running domain including DR, hot-pluggable PCI I/O assembly (HPCI) support, domain driver requests and events, and virtual console support. One dxs daemon is started on the main SC for each running domain.
This daemon (event front end) receives notification of events from various SMS daemons and forwards them to subscribed clients. With SMS 1.4, the only client that can subscribe is Sun Management Center software.
This daemon (environmental status monitoring daemon) provides monitoring of the environment conditions of Sun Fire systems, including system cabinet conditions and fan tray and power supply temperatures. One instance of the esmd is run on the main SC.
This daemon (failover management daemon) is the center of the SC failover mechanism. It will detect faults on remote or local SCs and takes appropriate action. One instance of fomd is run on the main and spare SCs. This daemon uses RPC services on the SCs and is the reason why rpcbind is not disabled.
This daemon (FRU access daemon) is the field-replaceable unit (FRU) access daemon for SMS. It is the mechanism by which access is provided to the serial electrically erasable programmable read-only memory (SEEPROMs) within the Sun Fire frame, to which the SCs have access. The frad is run on the main and spare SCs.
This daemon (hardware access daemon) implements hardware access for SMS daemons used by the daemons to control, access, configure, and monitor hardware. The hwad is run on the main and spare SCs.
This daemon (key management daemon) manages the IPsec authenticated communication between the SCs and domains. One instance of kmd is run on the main SC.
This daemon (management network daemon) supports the internal Management Network (MAN). The role played by the mand daemon is specified by fomd. One instance of mand is run on both the main and spare SCs.
This daemon (message logging daemon) accepts the output of all SMS daemons and processes and logs those messages based on its configuration files. The mld is run on the main and spare SCs.
This daemon (OpenBoot TM PROM support daemon) supports the OpenBoot PROM process running on a domain through the mailbox that resides on the domain. When the domain OpenBoot PROM writes requests to the mailbox, the osd daemon executes those requests. Only the main SC is responsible for booting domains. One instance of osd is run on the main SC.
This daemon (platform configuration database daemon) is responsible for managing and controlling access to platform and domain configuration information. The pcd is run only on the main SC.
This daemon (SMS startup daemon) starts, stops, and monitors all the key daemons and servers of SMS software. One instance of ssd is run on both the main and spare SCs.
This daemon (task management daemon) implements task management services for the SMS software such as scheduling. Currently, this daemon is used by setkeyswitch and other daemons to schedule hardware power-on self-test (HPOST) invocations. The main SC is responsible for these types of events, so one instance of tmd is run on the main SC.
SC Network Interfaces
There are several network interfaces used on an SC to communicate with the platform, domains, and other SCs. Most of these interfaces are defined as regular Ethernet network connections through /etc/hostname.* entries.
Main SC Network Interfaces
A typical main SC (sc0 in our sample) has two files in /etc with contents similar to the following:
# more /etc/hostname.scman0 192.168.103.1 netmask + private up # more /etc/hostname.scman1 192.168.103.33 netmask + private up
In addition, a typical main SC has corresponding entries in /etc/netmasks:
192.168.103.0 255.255.255.224 192.168.103.32 255.255.255.252
Non-routed (RFC 1918) IP addresses are used in all SC examples. We recommend that you use these types of IP addresses when deploying Sun Fire SCs. The SMS software defines internal SC network connections to be private and not advertised.
Domain-to-SC Communication (scman0) Interface
The /etc/hostname.scman0 entry sets up the I1 or domain-to-SC MAN. The first IP address in our example, 192.168.103.1, is controlled by the SMS software to be always available only on the main SC.
From a security perspective, misuse of or attacks on the I1 MAN network between the domains and the SCs may adversely impact domain separation. The hardware implementation of the I1 network within a Sun Fire 12K or 15K chassis addresses these concerns by permitting only SC-to-domain and domain-to-SC communication. The I1 MAN network is implemented as separate point-to-point physical network connections between the SCs and each of the nine domains supported by a Sun Fire 12K system or 18 domains supported by a Sun Fire 15K system. Each of these connections terminates at separate I/O boards on each domain and SC.
On the SCs, these multiple separate networks are consolidated into one meta-interface to simplify administration and management. The I1 MAN driver software performs this consolidation and enforces domain separation and failovers to redundant communication paths.
Direct communication between domains over the I1 network is not permitted by the hardware implementation of the I1 network. By implementing the network in this manner, each SC-to-domain network connection is physically isolated from other connections.
Although the scman0 network supports regular Internet Protocol (IP)-based network traffic, it should be used only by Sun Fire management traffic. Any other use of this internal network may affect the reliability, availability, serviceability, and security of the entire platform. Refer to the scman (7D) and dman (7D) man pages for more information.
SC-to-SC Communication (scman1) Interface
The /etc/hostname.scman1 entry is used to configure the I2 or SC-to-SC MAN. This network connection, on which both SCs have an IP address, is for the heartbeat connections between the two SCs.
Both of the I1 and I2 MAN network connections are implemented internally in the Sun Fire 12K or 15K chassis. No external wiring is used.
Spare SC Network Interfaces
The spare SC has the same physical network interfaces as the main SC. The scman0 network interface is plumbed by the Solaris OE through the /etc/hostname.scman0 file on the spare SC in the same manner, and with the same information as on the main SC. The difference between the main and spare SCs is that the interface is inactive on the spare. The spare SCs scman0 port on the I/O hubs is disabled and mand does not provide path information to scman0 on the spare.
The scman1 interface, which is for SC-to-SC communication, has the following configuration information for this interface:
# more /etc/hostname.scman1 192.168.103.34 netmask + private up
In addition, the spare SC has the following corresponding /etc/netmasks information:
Main and Spare Network Interface Sample Configurations
Use the following command to verify the status of the main SC:
# showfailover -r MAIN
Our network configuration sample appears as follows on the main SC (sc0):
# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.1.72.80 netmask fffff800 broadcast 10.1.79.255 ether 8:0:20:a8:db:2e scman0:flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3 inet 192.168.103.1 netmask ffffffe0 broadcast 192.168.103.31 ether 8:0:20:a8:db:2e scman1:flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4 inet 192.168.103.33 netmask fffffffc broadcast 192.168.103.35 ether 8:0:20:a8:db:2e
Although the scman0 network supports regular Internet Protocol (IP)-based network traffic, it should be used only by Sun Fire management traffic. Any other use of this internal network may affect the reliability, availability, and serviceability, and security of the entire platform. Refer to the scman (7D) and dman (7D) man pages for more information.
Our sample network configuration appears as follows on the spare SC (sc1):
# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.1.72.81 netmask ffffff00 broadcast 10.1.72.255 ether 8:0:20:a8:ba:c7 scman0:flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3 inet 192.168.103.1 netmask ffffffe0 broadcast 192.168.103.31 ether 8:0:20:a8:ba:c7 scman1: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4 inet 192.168.103.34 netmask fffffffc broadcast 192.168.103.35 ether 8:0:20:a8:ba:c7
Security Options in SMS 1.4
To improve network performance on the I1 MAN network, sequential MAC addresses are used by default on each of the up to 18 domains. With this configuration, it is straightforward to determine what the MAC address is of any given domain. It is possible, therefore, for a domain to broadcast gratuitous Address Resolution Protocol (ARP) information containing erroneous MAC addresses. The SCs accept these malicious MAC packets and use them to misroute packets destined for domains. To protect against this type of ARP spoofing attack and other IP-based attacks, two options are available with SMS 1.4:
Disable ARP on the I1 MAN network between the SCs and domains.
Disable all IP traffic between the SCs and a domain by excluding that domain from the SCs' MAN driver
We strongly recommend that you disable ARP on the MAN network in all multi-domain Sun Fire configurations. For multidomain system configurations where domain separation is of critical concern, we also recommend disabling IP connectivity between the SCs and all domains that require separation.
Disabling ARP on the MAN network provides some protection against ARP attacks, but it still leaves all other IP functionality present in the I1 network. If more stringent security is required, disabling all IP traffic between the SCs and one or more individual domains on the I1 network may be necessary. Instructions for implementing these two options are provided later in this article.
Disabling ARP on the I1 MAN network impacts all domains and SCs within a Sun Fire 12K or 15K chassis. Implementing this option requires modifications to the SCs and all domains. Additionally, this option requires rebooting all domains and the SCs.
We recommend running software on the SCs to monitor modifications to the ARP table, regardless of whether you use either of these options. One example of such software is arpwatch. Other similar software available either freely or commercially can be used. The freeware arpwatch software generates alerts based on ARP table modifications and is available from a variety of security sites on the Internet. As with any other non-SMS application to be run on an SC, the impact of arpwatch must be evaluated against the OpenSC guidelines presented in the Sun Fire 15K Open System Controller (OpenSC) white paper.
If a domain is excluded from the MAN network, the domain-to-SC network interface dman0 is not configured at installation time. Even if the dman0 interface is manually configured, the domain cannot communicate with the SCs because the domain is excluded from the SCs' perspective. This solution provides excellent protection for the Sun Fire 12K or 15K chassis against malicious domains attempting to attack either the SCs or other domains in the chassis. We recommend this solution for environments that require strongly enforced separation between domains and SCs.
Be aware that when you disable all IP traffic on the I1 MAN network, some functionality is no longer available. The services that are unavailable are as follows:
Dynamic reconfiguration (DR) from the SCs: commands such as addboard, removeboard, deleteboard, and rcfgadm cannot be used for domains excluded from the I1 MAN network
Network time protocol (NTP) from the SCs for the domains
The use of showdevices, from the SC, to display devices associated with domains
Domain-side DR is still available for domains that are excluded from the MAN network. Also, console access to the domains is available because console traffic does not have to use the internal I1 MAN network. Console access can use the Sun Fire 12K or 15K server's IOSRAM, or "mailbox," connection to reach the domains. The IOSRAM interface is not TCP/IP based. Services using the IOSRAM interface, such as domain booting, remain available even if IP traffic to one or more domains is disabled.
Ultimately, security policy and enterprise application requirements may be the deciding factor as to which option is most suitable. Disabling ARP on the I1 MAN network provides some protection for domains against ARP attacks, but it still leaves all the functionality present in the MAN network. If more stringent security is required, disable all IP traffic between the SCs and one or more individual domains on the MAN network.
To enforce strict separation between a domain and all other domains and SCs in a Sun Fire 12K or 15K chassis, we recommend that the domain be excluded from the MAN network. This change can only be performed on the SCs and is described later in this article.