Peachpit Press

Selecting a Secure Enterprise OS: Don't Make the First Step the Wrong Step

Date: Oct 28, 2005

Return to the article

It's pretty common to focus on functionality when choosing an operating system, particularly for businesses with specific technical needs. But Bruce Potter warns that making your selection without paying due attention to the operating system's security issues may hit hard in the long term.

Selecting an operating system for use in your enterprise can be a complicated decision. Licensing costs, supported software, hardware options, reliability, and current administration capabilities all are part of the equation. Security is also a concern, but sometimes it's difficult to determine what "security" really means with respect to selecting an operating system. Further, the major operating system choices have a great deal of marketing hype with respect to security, but it's hard to cut through the hype and make a decision that's best for your environment.

Ultimately, the choice you make in operating systems for your enterprise is a choice you'll have to live with for years. Migrating from one operating system to another can be an expensive proposition. So it's best to make your choice in an educated manner. Security, while maybe not your highest priority, is an aspect of the operating system that will certainly have long-term ramifications. This article provides one view of operating system security that I hope will help you in your decision. While I have my own opinion on what OS you should choose—I'll just tell you upfront that I'm a FreeBSD zealot—I'm going to try not to let my opinions get in the way of the facts at hand.

Operational Security

In order to understand which OS meets your needs, you must understand what "security" really means in an operational construct. Operational security is about the ability to maintain a secure and robust environment over the long term. It's not enough to have a single host system that's resilient to attack in the here and now; all the systems in the enterprise have to be able to stay secure as technologies, attacks, and applications change.

Anyone can be trained to lock down a host. There are host lockdown guidelines from the likes of Microsoft and NIST that are very well done and are easy to follow. Any reasonably savvy IT professional can follow these directions and make a system difficult to compromise. But ultimately, these procedures are really just part of the equation. If you have to constantly change your system configuration or don't have any idea of how the security of your operating system is going to change from version to version, then you still haven't achieved operational security.

The manner in which your operating system is developed can have a profound impact on your enterprise. Operating system development encompasses the entire lifecycle of the software. How it's planned, designed, implemented, tested, and maintained ultimately affect your environment, but the effects are a matter of debate. Some would say that a rigorous, structured process is the only way to create secure and scalable software. This is a very corporate view of software development and is commonly evangelized in complex systems development. However, others would argue that simply using a process doesn't mean that the software is really more secure—just that it was developed in a consistent manner.

Conversely, the open source world tends to take a different view of this problem. A large number of developers developing software in a relatively ad hoc manner puts more eyeballs on the code and forces only "important" things to be integrated. This development model may create software that has the correct feature set, but there's nothing inherent in it that addresses security concerns at a tactical or architectural level.

The three major operating systems in use in datacenters (Microsoft Windows, Linux, and FreeBSD) have three very different development models. Let's take a look at each and consider how the development process affects security.

Windows Development Model

Microsoft Windows is a commercial operating system developed by a company the size of a country. The vision for the OS is created at the top and pushed down to all parts of the organization. Further, this vision goes beyond the operating system; supporting administrative tools, common userland applications, and even productivity software are all developed in an integrated and coherent fashion. The core parts of the operating system share architectural design elements with software throughout the Microsoft product offering. For instance, the cryptographic service provider is leveraged by nearly every Microsoft product that needs cryptographic capability. Also, authentication information is shared throughout the levels of the OS and can be accessed in a uniform fashion by applications (see Figure 1).

Figure 1

Figure 1 Microsoft controls development of everything from the core kernel to server applications.

From a security perspective, not only is security functionality able to be accessed and architected in a common way, but Microsoft also controls the patching process. This is a critical part of operational security. Microsoft controls all patches for their operating system, server software, and productivity applications. Also, they are now releasing all their patches on a regular schedule (the second Tuesday of every month). This setup allows administrators to prepare to test and deploy security patches in a coherent manner. By not having ad hoc patching, Microsoft allows enterprises to plan security upgrades in advance and ultimately probably reduce downtime.

Finally, Microsoft controls the long-term security planning for the primary operating systems and all supporting applications. They have a long-term product roadmap that includes security enhancements and functionality. While a product roadmap doesn't necessarily mean a more secure system, Microsoft has a robust developer network. By having a product roadmap with prerelease software available to developers, it increases the likelihood that developers will be aware of new security functionality and use it properly. Further, Microsoft's developer tools natively support Windows security functionality.

FreeBSD Development Model

Unlike Windows, FreeBSD is not a commercial venture. FreeBSD is developed by a team of individuals who build the OS as a system. The FreeBSD kernel is released with a complete set of system utilities, drivers, and configuration files that represent a formal release. For instance, FreeBSD 5.2 was released with the kernel, a fully functional firewall, network drivers, administration utilities, and core userland tools. Other userland applications are supplied by third-party developers (for instance, Mozilla from the Mozilla Foundation or Osiris from Brian Wotring), but these programs are packaged by FreeBSD for inclusion in the operating system. During the packing process, the programs are tested by the FreeBSD team to make sure that they work properly with the operating system—but not necessarily for security issues.

Because FreeBSD is more than just a kernel, security functionality can be architected throughout the kernel and core system utilities as the operating system is developed. For instance, the FreeBSD kernel has a concept of secure levels. Different secure levels have differing restrictions. For instance, at secure level 2, file systems can't be mounted and time cannot be adjusted more than a second at a time. Not only does the kernel understand the secure levels, but core system utilities modify and help enforce the secure levels. This is possible because FreeBSD is developed as an end-to-end system (see Figure 2).

Figure 2

Figure 2 The FreeBSD Project controls the development of the kernel and the core systems utilities.

This release process is controlled by a release engineering team that determines which features will be included in what version of the OS and when the next FreeBSD is ready for release. The release engineering team has a release schedule that outlines the next several FreeBSD releases, as well as determining when old releases will reach their end of life (EOL). The EOL date is important to operations because it's the last date on which the FreeBSD will release patches for that version of the operating system. If you want to stay current (and secure), you have to update by the time the EOL hits for your operating systems.

Linux Development Model

Linux is a very different animal from FreeBSD. Even though they're both open source operating systems, the Linux development model is much less coordinated. Linux, at its core, is simply a kernel. The core system utilities, including many of the drivers and tools that administrators use every day, are created by a different community and chain of command than those of the core kernel (see Figure 3). The Linux kernel is a discrete entity that's developed outside direct input from the rest of the "Linux community." From a security standpoint, this means that security capability that's created in the userland utilities is not necessarily reflected in the core kernel (and vice versa).

Figure 3

Figure 3 Linux is only the kernel with distributions that package the systems utilities and applications.

The kernel and userland applications are then integrated by a distribution (such as Red Hat or Debian) to create a complete operating system. The distributions may change some of the kernel or userland code when the code is integrated together. Some of these changes are minor "glue" changes; others are major pieces of new functionality. Each distribution handles integration in its own fashion, and this effectively makes each distribution its own operating system. So, rather than thinking "Linux," you must think "Red Hat" or "Debian" or "Mandrake," etc. when you're determining operating systems for your enterprise.

With respect to release planning and security functionality, each distribution has its own schedule and technology roadmap. For a while, Red Hat maintained a security roadmap (basically, it consisted of "integrate SELinux into the core OS"), but that seems to have been abandoned. Similarly, each distribution controls its end-of-life process, so there is no single "Linux EOL."

Patching in Linux-based operating systems is a bit of a two-part process. When a vulnerability is discovered, the vulnerability is disclosed to the original maintainer of the code. The original maintainer then releases a patch for the vulnerable software. Each distribution must pick up the patch and create a custom patch specific to their code base. At this point, administrators can then apply the patches to their system. This two-stage patch process can cause delay in the patching process; more importantly, it can cause confusion and configuration management problems.

Parting Shots

I've painted a pretty bleak picture of Linux operational security. Due to its development model, it's difficult to determine the future of the operating system, and maintaining it can be a chore. However, Linux has a great number of features and outstanding programs that are very useful in many operations. So, while there may be a tradeoff between security and utility, Linux is often a tenable situation.

When choosing an operating system for use in your enterprise, you need to look beyond the security capabilities "on the box" and think about how the OS really affects you over the long haul. You need to be prepared for the patching, upgrading, and maintenance issues each OS has and understand how these activities affect your security and availability. Ultimately, the enterprise operating systems discussed in this article are very advanced pieces of software that enable businesses around the world to perform IT functions. What works for you may not be what works for your neighbor, but you need to be aware that security for each of these operating systems is a complex issue requiring as much diligence as any other software selection in your enterprise.

1301 Sansome Street, San Francisco, CA 94111