Managing Your Oracle Solaris Cluster Environment

This chapter is from the book

Oracle Solaris Cluster Essentials

Learn More Buy

This chapter is from the book

This chapter is from the book 

Oracle Solaris Cluster Essentials

Learn More Buy

Patching and Upgrading Your Cluster

The approach you take to patching and upgrading your cluster will vary depending on your data center standards. Some data centers apply only the bare minimum of changes, such as critical security patches, operating on the principle of "if it ain't broke, don't fix it." Clearly, if a system is stable, this is a reasonable approach, especially if availability is your highest priority. Other data centers schedule regular planned outages to perform patching, often choosing to lag a little behind the most recent patches to avoid encountering bad patches. Still others take the "latest and greatest" approach, given that the patches are meant to fix all the key, known issues. Whichever strategy you use, minimizing the overall outage will clearly be your main priority.

You can also upgrade the Solaris Cluster software by patching it with the patches that are built into a later release of the Solaris Cluster software. However, you must follow all the instructions in the README file contained within each patch.

Upgrade Methods

Three main methods are available for patching or upgrading your operating system and Solaris Cluster software: standard upgrade, rolling upgrade, and dual-partition upgrade. Solaris Live Upgrade provides yet more options when used with these methods. Each method results in differing lengths of outage on your cluster, which further depends on the number and type of zones you have configured (see the section "Oracle Solaris Zones" in Chapter 3, "Combining Virtualization Technologies with Oracle Solaris Cluster Software"). The following sections describe these methods.

Before you begin making changes to the system, consider the following checklist:

• Has the upgrade procedure been tested on a test cluster before being implemented on the production system? If not, is management aware of the potential risk of untested procedures?
• Have you verified that the new configuration is supported by all your services? Some applications place greater restrictions on the operating system or other software versions they support.
• Have the proposed changes been agreed upon and recorded in your change control system?
• Do you have a full backup of the cluster prior to making any changes?
• Have you tested that you can switch over all services between cluster nodes prior to your changes? This testing isolates any post-upgrade problems to the upgrade and patches.
• Is there an agreed outage period of a set length?
• Is there a procedure to reverse any changes if the upgrade does not go according to plan? Have you determined when you will need to start backing out any changes if that happens?
• Have you tested that you can switch over all services between cluster nodes after the changes have been made? This testing ensures that if a failure occurs later, your system will continue to provide a set of highly available services.

Standard Upgrade

With standard upgrade, you schedule a complete cluster outage to perform the patch or upgrade process. This method might be considered for applying patches, such as kernel patches, that need to be installed in single-user mode. Major operating system upgrades also fall into this category. You cannot run a Solaris Cluster configuration with nodes running different major versions of the Oracle Solaris OS, for example, Solaris 9 OS and Solaris 10 OS.

A standard upgrade is by far the simplest method but necessitates the longest cluster outage. During this period, your services are unavailable. Consequently, this method is the least desirable from an availability standpoint.

A standard upgrade can be combined with Solaris Live Upgrade. This approach significantly reduces the cluster outage by no longer requiring the entire cluster to be shut down while you are upgrading the software. You first create an alternative boot environment with the upgraded software on each machine (see "Solaris Live Upgrade" for details). When you are ready to activate the new software, halt the entire cluster, and then boot the entire cluster using the alternative boot environment with the new software on each machine.

Rolling Upgrade

With rolling upgrade, the nodes in your cluster are upgraded one at a time. Any service hosted on a node to be upgraded is manually switched over to another cluster node prior to the target node being shut down for patching. After the target node has been patched, it is brought back into the cluster, and the process is repeated until all nodes have been upgraded. After all of the nodes have been upgraded, you execute the scversions -c command to commit the cluster to use the new software. The cluster software does not enable cluster-wide protocols until the commit happens. Prior to the commit, all nodes, including nodes with new software, continue to use the old cluster-wide protocols. There is no rolling downgrade after the commit happens. The outage incurred by a service is therefore just twice the time it takes to switch over from one cluster node to another. A switchover of a simple application can take about 30 seconds, though the time varies significantly depending on the application.

A rolling upgrade changes only the Solaris Cluster software or the Oracle Solaris OS. Rolling upgrades can be used as long as the major cluster release and major operating system remain the same, for example, the Solaris Cluster 3.2 software or the Solaris 10 OS. You cannot perform rolling upgrades between major cluster or major OS releases, for example, the Sun Cluster 3.1 software to the Solaris Cluster 3.2 software or the Solaris 9 OS to the Solaris 10 OS.

The rolling upgrade process updates the software on the local storage of a node when that node is not in cluster mode. The highly available (HA) container places the software on shared storage. Because the shared storage is never out of service, the rolling upgrade process has no opportunity to change the software residing on shared storage. Therefore, a rolling upgrade cannot be used to upgrade software on HA containers.

A rolling upgrade can also be combined with Solaris Live Upgrade. This combination does not reduce the service outage. A rolling upgrade without Solaris Live Upgrade reduces the service outage by performing software upgrades while the machines are not in the cluster. During this time period, the ability of the cluster to survive machine failures is reduced. The use of Solaris Live Upgrade can significantly reduce the time period of increased vulnerability. Solaris Live Upgrade also reduces the period of time in which nodes in the cluster have different software versions. You first create an alternative boot environment with the upgraded software on each machine (see "Solaris Live Upgrade"). You then serially halt each machine and boot that machine from the alternative boot environment with the upgraded software. After you have booted all nodes from the alternative boot environment with the upgraded software, you execute the scversions -c command to commit the cluster to use the new software.

Dual-Partition Upgrade

A dual-partition upgrade offers the greatest flexibility in terms of what changes you can make to the system. The change can be a patch, an update release, or a major release. Changes in multiple areas can be applied simultaneously. A dual-partition upgrade can even be used when all the changes are in areas other than the Solaris Cluster software, including the following:

• The Oracle Solaris OS
• A Solaris Cluster software release
• Third-party volume managers and file systems
• Application software (as long as the applications are not installed on shared storage)

A dual-partition upgrade divides your cluster into two parts, each of which is updated separately and sequentially. To invoke a dual-partition upgrade, you must use the scinstall command from the target media of the release to which you are upgrading. This command then prompts you with the options shown in Example 4.11. Choose option 3 for a dual-partition upgrade. You must divide the cluster machines into two sets of nodes, which are called partitions. You must partition the cluster such that each partition has for each storage device at least one node that can access that storage device. During the upgrade process, each partition will host the cluster applications, and the cluster applications must be able to access storage. If you are unsure of how to partition the cluster, you can choose an option whereby the system presents you with the various ways to group your cluster nodes for the purposes of the upgrade. After you have chosen your partitioning, you next select the option to initiate the actual upgrade.

Example 4.11. Main Menu of the scinstall Command Located on the Solaris Cluster Release Media

The main menu of the scinstall command provides you with the following options:

*** Main Menu ***

    Please select from one of the following (*) options:

        1) Create a new cluster or add a cluster node
        2) Configure a cluster to be JumpStarted from this install server
      * 3) Manage a dual-partition upgrade
      * 4) Upgrade this cluster node
      * 5) Print release information for this cluster node

      * ?) Help with menu options
      * q) Quit

    Option:  3

The system software makes changes to the quorum votes, migrates your services to the partition that is still running while the first partition is upgraded, and halts the nodes of the first partition. You boot the nodes of the first partition into non-cluster mode and upgrade the software. After you have completed the upgrade on the first partition, you choose to continue the dual-partition upgrade. The system software reboots the nodes of the first partition into cluster mode as a new cluster. Unlike what happens in an ordinary boot, the nodes of the first partition proceed through the boot process and stop just prior to the point where the cluster would activate resources used by applications (such as mounting file systems and plumbing IP addresses) and start applications. At this point, most of the boot process has completed. The system software halts the nodes of the second partition and waits for them to actually halt. Next, the system software imports volumes, mounts file systems, plumbs IP addresses, and starts applications. In effect, the system performs a switchover of all applications and their resources from the second partition to the new partition. Next, reboot each node of the second partition and upgrade the software. When done with that step, you choose to continue the upgrade. The system software reboots the nodes of the second partition into cluster mode. After all nodes are back in the cluster, the system software restores the quorum configuration and completes the dual-partition upgrade.

If the dual-partition upgrade fails, you boot all nodes in non-cluster mode and execute the scinstall -u recover command on each node to restore the initial copy of the Cluster Configuration Repository (CCR). The system is already completely shut down at this point, so a minimal downtime upgrade is not possible. You do not restart the dual-partition upgrade. Instead, you perform a standard upgrade.

The service outage when you perform a dual-partition upgrade on a system with failover application is approximately the same as for a rolling upgrade. Because a time comes when neither partition can host an instance of a scalable application, there is a service outage for the scalable application, and the service outage is approximately the same as that of a failover application. For the simplest scalable applications, the service outage can be about 30 seconds, though the time varies significantly based on the application.

You cannot upgrade any software or data on the shared storage because the services remain running throughout the upgrade process, except for the point when the services are switched over from the partition running the old software to the partition running the new software. Do not make any configuration changes during the dual-partition upgrade. Make configuration changes either before or after the dual-partition upgrade.

A dual-partition upgrade can also be combined with Solaris Live Upgrade. This combination does not reduce the service outage. A dual-partition upgrade without Solaris Live Upgrade reduces the service outage by performing software upgrades while the machines are not in the cluster. During this time period the ability of the cluster to survive machine failures is reduced. The use of Solaris Live Upgrade can significantly reduce the time period of increased vulnerability. You first create an alternative boot environment with the upgraded software on each machine (see "Solaris Live Upgrade"). You then initiate the actual dual-partition upgrade. The system software automatically manages all of the work of the dual-partition upgrade. There no longer is any need to boot into non-cluster mode, so the number of reboots is reduced by half. The net result is that the time required for upgrades is dramatically reduced.

Choosing Which Upgrade Method to Use

The choice of an upgrade method is governed by three factors: the simplicity of the upgrade process, the compatibility of the current and target software environments, and the level of availability your services must achieve during the upgrade process. Thus, if you require simplicity above all else, then the standard upgrade is the best method. If your current and target software releases do not span a major release, and you want to maintain service availability even for scalable services, then you can use a rolling upgrade. Finally, if changes to your cluster environment involve software spanning major releases and you need to minimize your service outage, then you must choose a dual-partition upgrade.

Solaris Live Upgrade

Solaris Live Upgrade is not a stand-alone technology for upgrading a Solaris Cluster system. Solaris Live Upgrade must be used as part of either a standard upgrade, a rolling upgrade, or a dual-partition upgrade. Solaris Live Upgrade makes it possible to create an alternate boot environment with the upgraded software while the machine continues to run unaffected using the current boot environment. See "Standard Upgrade," "Rolling Upgrade," or "Dual-Partition Upgrade" for details on when to bring the alternate boot environment with the updated software into operational status. The boot environment with the old software is not damaged when you boot the alternate boot environment with the new software. If you encounter a problem with the new software, you can recover by halting all nodes of the cluster and then booting the entire cluster from the boot environment with the old software and configuration information. Solaris Live Upgrade is available for all the operating system releases supported by the Solaris Cluster 3.2 software.

If you are using the Solaris 10 OS with a ZFS root (/) file system, then the effort involved in creating an alternate boot environment is virtually zero because the required new file systems can be created from the existing root zpool. However, if you are using a Solaris Volume Manager or a Veritas Volume Manager mirrored root disk, then the process is significantly more complicated because you must find additional partitions to match the existing root partition allocation. Consequently, using the Solaris 10 OS and ZFS for the root disk has significant advantages. However, do not upgrade the software version of the zpools in your cluster until you are satisfied that your new environment is working correctly. If you do upgrade the zpool version, you will not be able to fail back to your old boot environment, as the old environment might not be able to mount the zpools that already have a higher version number. Such zpools cannot be downgraded.

After you have updated the Oracle Solaris OS on your new boot environment, you can proceed with updating the Solaris Cluster software. When using Solaris Live Upgrade, you must follow the documentation on how to handle specific steps relating to the volume managers, Solaris Volume Manager and Veritas Volume Manager, and which version of the Java Runtime Environment (JRE) is required.

The process for updating the Solaris Cluster software on the alternate boot environment is outlined in the following steps:

1. Create an installer program state file to drive the upgrade of the shared components on the alternate boot environment, but without actually performing the actions.
2. Run the installer program again using this state file but targeted on the alternate boot environment.
3. Update the Solaris Cluster framework by running the scinstall command from the new Solaris Cluster media, again targeting the alternate boot environment.
4. Update the Solaris Cluster agents by running the scinstall command from the new Solaris Cluster media, again targeting the alternate boot environment.

Upgrading Nodes Using Oracle Solaris Zones

When upgrading a cluster node that uses the Solaris 10 OS, you need to consider the impact the upgrade process has on any Oracle Solaris Zones (also known as Oracle Solaris Containers) that are configured on the system. These containers represent additional, virtual Solaris instances that must also be upgraded.

To maintain service availability during an upgrade, a resource group must have a node list that contains at least one physical cluster node that will still be available to host the resource group while the rolling upgrade or dual-partition upgrade is performed. This is feasible both for resource groups that use global-cluster non-voting nodes and for resource groups existing in zone clusters. However, HA containers are, by their very nature, the sole place in which the service resides. Therefore, taking the cluster node into single-user mode to patch it results in the HA container, and consequently the service, being unavailable for the duration of the update process [ZonePatch].

One way to minimize HA container service outage is to use Solaris Live Upgrade. The procedure is not particularly practical if you have tens or hundreds of HA containers because it involves many steps.

The following procedure is meant for environments where each HA container has its zonepath, on its own Oracle Solaris ZFS file system, in its own zpool, for example, ora-pool/ora1 for ora1. Both ora-pool and ora-pool/ora1 must have appropriate mount points set.

Solaris Live Upgrade requires full control over the zpools when the Oracle Solaris OS reboots in order to perform the necessary renaming of the ZFS file systems and the changes to the zones' configuration files. Therefore, the ZFS file systems must not be under Solaris Cluster software control during the shutdown and boot process. This can be achieved by creating an alternate boot environment where each zone and its associated zonepath are under cluster control. Then, prior to shutting down the system, you can offline and unmanage the resource groups controlling the HA containers and import each zpool that holds a zonepath. Finally, you can copy all the affected configuration files in the CCR and the zpool.cache file to the alternate boot environment.

In general, this procedure requires you to perform the following steps:

1. Alter the HA container's resource group node list to just the current node.
2. Perform a Solaris Live Upgrade on the node.
3. Disable the resources of the HA container's resource group, which takes the container offline and unmanages the resource groups.
4. Import the zpool holding the zone's zonepath.
5. Copy the zpool.cache file and corresponding CCR files for the resource group to the alternate boot environment.
6. Shut down and reboot the node.
7. After Solaris Live Upgrade has performed all of the necessary reconfigurations, manage and bring the resource group online.
8. Detach the HA container from its other nodes, and then use Solaris Live Upgrade and reboot the other nodes in turn.
9. Attach the zonepath of the HA container with the -F (force) option to bring the container's state back to installed.
10. Reset the HA container's resource group node list.
11. Check that switchover still works and finish by switching the HA container back to its primary machine.

Although this procedure requires a shorter outage, other factors might make it impractical. First, the HA container cannot be failed over while the upgrade is under way because the node list has been restricted to the single node. Additionally, the service-level agreements (SLAs) for other HA containers that are deployed on the system might limit when the outage can occur because any upgrade will affect them, too. If there isn't a single time period when groups of containers can be updated, then the restriction will prevent any of the changes from being made.

The final option for upgrading HA containers is to use the update on attach feature in the Solaris 10 10/08 release. This option can be used only to update the packages in a container and does not allow them to be downgraded. If you use this option, you must ensure that the process does not introduce any packages that are older than the current version the Solaris Cluster software requires. If it does, you must update them again using the installer program.

An outline of the steps for using the update on attach feature follows:

1. Evacuate (or switch over) all the containers that will not be patched from the target node (nodeT) using clresourcegroup switch ..., leaving just the containers to be patched.
2. On node nodeT, use zoneadm -z ... detach to detach each remaining container from nodeT so that the operating system will not attempt to patch them.
3. On node nodeT, patch nodeT and reboot back into the cluster.
4. Use clresource disable ... to disable the HA container resources that were switched over to the alternative node (nodeA), and switch their resource groups back to nodeT. This switches over the zonepaths to nodeT.
5. On node nodeA, detach the container from nodeA so that the operating system will not attempt to patch the container during the patch process or reboot the container after the upgrade.
6. On node nodeT, use zoneadm -z ... attach -u to reattach and patch the HA container on nodeT.
7. Test the containers by booting and halting them manually.
8. Place the containers back under cluster control by reenabling the HA container resource.
9. Repeat the upgrade process with nodeA.

Ensure that HA containers are prevented from failing between the nodes while this procedure is being followed.