- Management and Deployment Data Flow
- Management and Deployment Authentication
- Security of Management and Deployment Network
- Security Issues during Management and Deployment
- Conclusion
This chapter is from the book
This chapter is from the book
This chapter is from the book
Management and Deployment Authentication
Now that you understand how the data flows around the management network, we should discuss how each of these data paths is authenticated. It is possible that there could be many authentication methods to gain access to the same set of data. If multiple paths exist to the same sets of data and multiple authentication and authorization paths into that data, then security issues could exist that are currently unknown. The most prevalent security issue is inadvertent information leakage. For example, if one user was denied access within the VIC to see a VMs data within VC but is then allowed to see this data within LifeCycle Manager, this could be a potential for information leakage and unauthorized access. The claim could be made that this information about the VM is trivial—the name of the VM and its virtual hardware makeup, among other things. However, if this VM is a classified VM, none of that information would normally be seen by anyone not within the appropriate classification. Therefore, this example would provide a breach in security because of information leakage.
This possibility is the main reason nearly all the current security guidelines dictate that some form of directory service be employed to control authentication and authorizations.
Difference Between Authorization and Authentication
There is a huge difference between authorization and authentication. Authentication is the act of confirming the identity of the user. Authorization is to what that identity has the right to access. In VC terms, authorization is the roles and permissions assigned to a user. Yet roles and permissions exist for VC and the VMware ESX or ESXi hosts. Roles and permissions do not translate to direct access to the management appliance through other means. Roles and permissions definitely do not expand to include all the other management tools. This often creates a split-brained authentication and authorization situation, even when directory services are in use, because several sets of roles and permissions are possibly in use. This leads to confusion at the very least!
Split-Brain Authentication
Split-brain authentication occurs when there is more than one method to authenticate a user to a given role. Each part of the VMware Virtual Environment has its own authentication method. Let's look at the special example of the administrative user. The administrative user for VMware vCenter is the user named administrator by default, whereas for VMware ESX and ESXi, the administrative user is named root. For VMware Server, root is used when VMware Server is hosted on a Linux system and administrator is used when VMware Server is hosted on a Microsoft Windows system. Each of the other management tools, such as LifeCycle Manager, uses a different default administrative user.
This causes quite a bit of confusion in most cases. Different usernames imply that first there needs to be a mapping between the users, which includes the authorizations for the user that we will discuss in the next section. The tool often used to mitigate this confusion is the use of a directory service like active directory, LDAP, NIS, eDirectory, and so on. This implies that all systems within the virtual environment should also use the same directory service. However, it is impossible to use the same directory service everywhere and even if you did, the directory service could not be used for any user for VMware ESXi or the root user for VMware ESX and VMware ESXi.
You may wonder why it is not possible to override the root user, as we can set any user within the directory service to be part of an administrative group and be granted the proper authorizations on Microsoft Windows systems. The answer is fourfold. First, root is just a name for a user with a user id of 0. Any user with a user id of 0 is in effect a root user. It is not possible to change the definition of userid 0. Second, if you are hosting your directory service within a VM and you need to boot that VM, you may have to log in to the system as the root user to start the VM whether this is via SSH, using the VIC, or using the console. If you cannot authenticate the root user because its authentication is direct to the directory server, you cannot start the VM. Unlike Microsoft Windows, where credentials can be cached locally for direct login, with GNU/Linux the credentials cannot be cached, so a chicken and the egg situation may exist. Note, however, that the VIC and RCLI do not use cached credentials. The third reason is that with VMware ESXi version 3.5, it is impossible to set up a directory service on the system because the capability does not exist within this version of the product.
The fourth reason is that to give a user these privileges, you must in effect make the user root (with a user id of 0). This is frowned upon because no audit trail exists in this situation. Instead, we use other tools to gain this audit trail and remove multiple root accounts, which can increase the possible attack surface of the virtualization host. In addition, improperly set up directory service access can also lead to an increase in the possible attack surface of the virtualization host.
So there can be local accounts as well as directory service accounts, and at least one local account should be available in case the directory service authentication fails for some reason. There can also be users with local accounts as well as directory service accounts. This leads to multiple forms of authentication for a given user and therefore split-brain authentication.
Split-Brain Authorization
Of the two, split-brain authentication and authorization, the worse problem is split-brain authorization, or the capability for one system in the virtual environment to allow access to data that other systems do not allow. With so many management tools, it is quite possible that one of the tools will allow access to data that would not be available via another tool. I will highlight these issues with a few examples.
A user who knows an administrator user and password, but not necessarily the root password of a VMware ESX or VMware ESXi host, can directly attach the VIC to the host. By doing so they are presented with the capability of adding more users directly to the hosts. This capability is not normally available if you use the VIC directly connected to VC. However, note that unless the administrator makes a change within the permissions for VIC to VMware ESX or ESXi host, it is impossible to log in using a user that is not root. This use has two views of the same virtual environment, and one provides the chance for further abuse of the system by being able to create new users.
Another example of split-brain authorization is a user who can approve virtual machines within LifeCycle manager; yet when the user logs in to vCenter, the user can also create virtual machines within the environment. Should the approver also be the creator of the virtual machines? If so, this can also lead to abuse because the approver has the rights to approve VMs in one system, which automatically create the VMs for the requestor. Yet the approver can bypass the approval stage and create his own VMs without a record of the VM being created. In this example, the user once more has two views for the same data and more privileges within one than the other system. The approver may need only read-only access to check on performance and other metrics but not need to be able to do anything else within the system.
When all management tools are in use, it is important to have very well-defined roles for all users and then assign the appropriate authorizations for each user across the virtual environment. This requires you to create a mapping from one management tool to another because not all tools have the same names for each role or permission.
Mitigating Split-Brain Authorization and Authentication
Several items must be implemented to mitigate split-brain authentication and authorization. First, use a directory service for all non-emergency users. Second, create well-defined roles for each user. Third, enable remote logging for auditing of possible violations of these roles. In some cases even the use of a directory service will not be allowed specifically for VMware ESXi v3.5 and Web applications that do not contain directory service integration. In addition, with VMware ESX, VMware ESXi version 3.5, and VMware Server, it is possible to have multiple local accounts that mimic administrator or domain user accounts that could use different credentials, even with a directory service in play. The use of a directory service can mitigate split-brain authentication for all but the emergency use accounts that should never be overridden. Yet, the emergency use accounts should still have an audit trail, so on VMware ESX, ESXi, and VMware Server hosts, this emergency account should never be the users root or administrator but some other account that can access the proper commands while providing an audit trail.
Several audit trails are available to virtualization servers, but for VMware ESX or Linux-based VMware Server hosts, the most widely used command to provide this audit trail is the /usr/bin/sudo command with its log file of issued commands within /var/log/secure. VMware ESXi does not have this command because direct access to the management appliance console is not a suggested practice. Use of the VIC or RCLI is the suggested practice, which leads us to an audit trail that is also available for VMware ESXi. This is the use of the hostd log file as an audit trail, which is located within the /var/log/vmware directory. This log file, as well as others, can be sent to a remote logging host using changes to the syslog daemon configuration. Use of a remote logging host is the recommended way of maintaining an audit trail because attackers have been known to remove their entries from the local log files to hide their tracks. A remote logging server presents yet another system to which they need access in order to hide their tracks. You may even go so far as to further firewall the logging server from your virtual environment to further prevent attacks. However, this is outside the scope of the book.
The preceding discussion on auditing is Linux-centric, but remote logging is also available for Microsoft Windows-based VMware Server hosts, as well as VMware VirtualCenter, and those Linux and Microsoft Windows based workstations where the VIC, RCLI, and VI SDK are used and launched. An audit trail should tell you when, where, who, and how the system was changed, as well as provide a way to repeat all actions and get the same result. There is quite a bit to setting up a proper audit trail within the virtual environment.
The unfortunate truth is that log files are not generally small things, and in even a small environment you can very shortly have gathered gigabytes of data for review. No one has time to do this review by hand, so it is best to use some form of tool that will do this for you and notify you of issues that should be brought to your attention. I have used the Linux-based tool named logcheck, but there are many other tools to use. The key to using one of these tools is to train it to ignore those items that you have no need to view on a regular basis and those items that are purely informational. You also want it to have the capability to send email or other notifications based on the severity of the issue found. A host of these tools are available for both Linux and Microsoft Windows hosts.
It should be noted that such a tool is often required by Sarbanes-Oxley and other compliance standards to which many companies now adhere, so you may already have something in use within your environment.
Setting Up Microsoft Windows Systems for Remote Logging
Microsoft Windows does not have built-in methods to log items that appear within log files or the Event Viewer directly to a remote logging server. To perform this action, you need to use third-party tools. There are several from which to choose, however. When you are choosing a remote logging tool for Windows, you need to be sure that it can remotely log all entries that you see within the event viewer, as well as the specific text-based logs for VMware vCenter, Lab Manager, Site Manager, and LifeCycle Manager. Another thing to consider for your Windows systems is what to log. A good place to start your search could be the open source Snare project at www.intersectalliance.com/projects/index.html. Snare provides the Snare Agent for Windows and Snare Epilog for logging general text log files to a remote syslog server.
Because there is no tool specific to the Microsoft Windows platform, the setup is outside the scope of this book. However, from where to log is not outside of the scope of this book.
You will want to remotely capture event and other text logs from your VC, Lab, Site, LifeCycle, and Update Manager hosts as well as from whichever Microsoft Windows workstation you launch the VIC, RCLI, or VI SDK applications. Depending on the application, you may also want to remotely log local log files.
It is also recommended that you put a logging wrapper around all access to the RCLI. This way, you will know what commands were issued with which arguments. Such a wrapper should not display passwords if any exist within the logs. The reason for creating this wrapper is that not all actions will be logged by the VMware ESX or ESXi hosts.
For VirtualCenter, enable remote logging of all files within C:\Documents and Settings\All users\Application Data\VMware\VMware VirtualCenter\Logs. This is a very good directory to watch for log files. Unfortunately, unlike the VMware ESX or Linux logs, the logs increase in number, so you may have to specify a directory when using remote logging with VC.
Setting Up VMware ESX for Remote Logging
Before I explain how to set up remote logging, we must first decide what to log remotely. On a VMware ESX or VMware ESXi running on a Linux host, there are a number of useful logs. Remote logging would encompass, minimally, the following log files.
/var/log/vmkernel
/var/log/secure
/var/log/messages
/var/log/vmware/*.log
/var/log/vmware/aam/{*.log,*/*.log}
/var/log/vmware/aam/{*.err,*.out,*/*.err,*/*.out}
/var/log/vmware/webAccess/*.log
/var/log/vmware/vpx/vpxa.log
/vmfs/volumes/*/*/vmware.log
To enable remote syslog support on your VMware ESX or host modify /etc/syslog.conf and add the following line to the file.
*.* @remotehost
Location of this line is unimportant. It would be best if the remote host could be resolved using the /etc/hosts file over DNS, just in case DNS is not available. This also alleviates ARP cache DNS style attacks for remote logging. Unfortunately the RCLI does not set this up for VMware ESX.
The difficult part is to now get all the log files that syslog does not know about to go over the wire to the remote log server. This is where the Snare Epilog tool for Linux comes in handy. There is no default mechanism in place to perform this type of logging shipping with VMware ESX.
Setting Up VMware ESXi for Remote Logging
For ESXi there are several means to enable remote logging. One is via the graphical interface and the other is by using the RCLI. For the RCLI, you use the following:
vicfg-syslog --server ESXiServerName --username root --password password -- setserver remotehost --port remotehostport
If the remotehost is running a version of syslog then the --port remotehostport option will not be necessary. Unfortunately, if the logs are not collected by syslog on an ESXi system, no current supported mechanism exists to get those logs to appear on a remote logging host because it is not currently simple to install third-party software that will survive a reboot when using ESXi. In addition, the third-party software must be specifically coded for ESXi. At the moment no such tools exist.
Directory Services
One of the major tools used to alleviate split-brain authentication and authorization is a directory service such as active directory (AD), lightweight directory access protocol secure (LDAP-S), Novell eDirectory, or network information services (NIS). However, for directory services to mitigate this possibility, it is important that all management tools and hosts involved also participate within directory services.
One notable exception to this is VMware ESXi, which does not support directory services natively. However, all the tools do support them as long as you are first connecting to vCenter and not a direct connection to the host. However, as we all know, direct connection to the host is often required, specifically when we run the preceding command to control remote logging for VMware ESXi. So although use of a directory service mitigates many aspects of split-brain authentication and authorization, it is not a 100% surefire solution. There is no complete solution at the moment.
Configuring the VC to Start if Directory Services Are Not Available
In some cases when you virtualize your directory server, or when your directory server is not available, you will first have to boot your vCenter server. Normally you cannot do this if the directory service is not available. To mitigate this possible chicken and the egg situation, add the following lines near the end of the file C:\Documents and Settings\All users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg prior to the closing </config>.
<security> <ignoreUserResolveFailures>true</ignoreUserResolveFailures> </security>
This change requires you to restart the VMware vCenter service for it to take effect. Look within the log files within the directory C:\Documents and Settings\All users\Application Data\VMware\VMware VirtualCenter\Logs to look for any errors after changing anything within the vpxd.cfg file, because it will report on any errors within the log file.
Setting Up Directory Services on VMware ESX
There are many articles on the use and configuration of directory services within VMware ESX and Linux-based VMware Server distributions. Although much is written about the way these services are configured, there is not much on how to secure them. Also, there are many levels of integration with directory services. Some require more manual maintenance than others.
The integration methods all lack the capability to translate user login restriction using a group policy object set within the directory service. Some of the more manual modes do not require this group policy translation, because you are forced to use local logins for each user you want to access the system. However, that can be quite laborious to maintain on more than a few systems, and it pretty much ignores one of the major strengths of using a directory service: control of authorizations as well as authentications. There are a few issues to clear up before we implement any directory service.
First, do not set up directory service authentication for any user with a user ID less than 500, because these are the system users, and they must remain untouched for the system to run properly. A user ID is a unique integer assigned to all users, and this number is used internally, not the name associated with the number such as we described previously about the root user. Second, never set up directory service authentication for the root user, because that is your emergency login in case directory service is broken for some reason. In addition, you should never log in directly as the root user unless it is an emergency. Third, if you have to create user accounts on a system to finish the directory service integration, those user accounts could be the source of an attack if directory services fail, because the passwords default to those set on the system.
The last issue is the most important reason why partial directory service integration is frowned upon. It is also why you need to set up access policies to deny those users who are not in the proper groups, regardless of authentication success. One simple way to mitigate this is to make sure the local users are not in any special groups and to allow access only if you are logging in using a specific group.
The quickest way to implement this is to use the pam_access module for VMware ESX authentication and authorization. To do this, you need to follow some very basic steps.
- Add the following line to /etc/pam.d/system-auth.
account [default=bad success=ok user_unknown=ignore] /lib/security/ $ISA/pam_access.so
- Modify the file /etc/security/access.conf to reflect your group login policy, which will limit who can log in to only those within the given group. You can add the appropriate lines to the end of the file. Note this file is order dependent; you would not want to deny all access as the first line. Do not copy this verbatim; it is just an example explained afterward.
+:root:crond console -:ALL EXCEPT root:vc/1 +:GROUPNAME: NETWORK/NETMASK +:GROUPNAME: IP1 IP2 IP3 -:ALL:ALL
In this example, we are denying root access to the cron daemon, crond, as well as the console. Next we are disallowing root access to all but virtual console 1 (vc/1), which is accessed using ALT+F1. Next we are allowing all users in the group GROUPNAME to log in as long as they are either on the network defined by NETWORK/NETMASK or from either of the IP addresses: IP1, IP2, or IP3. Last, we deny access to all other users and groups from all other locations. The /etc/security/access.conf file can be as complex or as simple as you desire. In general, and at a bare minimum, you will want to disallow logins unless they are coming from users within the appropriate group and from the appropriate network or IP addresses. For more detailed information, use man access.conf from any Linux system or your VMware ESX service console.
However, this works only for login style attempts. There are other ways to control what additional services a person can access from the network. Unfortunately, these other methods do not know about groups or users, so they are not a part of directory service integration and thus will not be discussed here.
Many VMware ESX installations do not allow installation of third-party packages from RPM repositories. If this is the case, your ability to integrate with directory services will be hampered It is possible to do so, but testing of the integration will suffer, as well as future problem determination. Going forward, be sure to test on a development box before applying to a production server.
Integration with NIS
For those who are *NIX centric, VMware ESX can integrate with NIS to provide directory service functionality. The steps to enable this functionality follow.
- Use the following command from the service console (SC) command-line interface (CLI).
esxcfg-auth --enablenis --nisdomain=NISDOMAIN --nisserver=IPofNISServe
- Modify /etc/nsswitch.conf to look like the following. The main changes are to add the keyword nis to the group and shadow lines, which may not be there by default but must be there for full integration.
# Autogenerated by esxcfg-auth aliases: files nisplus automount: files nisplus nis bootparams: nisplus [NOTFOUND=return] files ethers: files group: files nis hosts: files dns nis netgroup: nisplus netmasks: files networks: files passwd: files nis protocols: files nis publickey: nisplus rpc: files services: files nis shadow: files
- Test to be sure everything shows up as expected. The following should show your normal password file contents plus any other users shared out by NIS. The group command will list your groups based on NIS as well.
getent passwd getent group
- Test to be sure NIS is working using NIS commands. The following commands will list the NIS specific users and groups. Note that if the third command does not return anything, then netgroup support does not exist on your NIS server. Investigate this with your NIS administrators because it will help with setting up the /etc/security/access.conf for the necessary pam_access configuration discussed previously.
ypcat passwd.byname ypcat group.byname ypcat netgroup
Partial Integration with Active Directory, LDAP, or LDAP-S
For those who do not implement NIS, other avenues exist for setting up VMware ESX hosts to use directory services. The three most popular are to use AD, LDAP, or LDAP-S. It is recommended that you never use LDAP for authentication because it is a clear-text or unsecured protocol. So we will not discuss this within this section. Partial integration implies that to complete the integration, you will need to manage user accounts per a VMware ESX host, and only the credentials and groups are handled via directory services. This is overall somewhat more secure than other methods, because if you do not have a login on the host there is no way to gain shell access. However, this has the drawback of requiring users to be maintained on all VMware ESX hosts. If you have more than a few hosts, this maintenance becomes a significant issue; you will need to remove accounts or add accounts when users leave or join the virtualization administration team.
Partial Integration with AD
Partial integration with AD requires the running of a simple set of commands. However, to test things you will need to add a new package to your system. This package is the krb5-workstation RPM, which you can find at any CentOS-3 or Red Hat Enterprise Linux 3 repository of packages. After you have it downloaded, install from the SC CLI using the following line. Note the version is relatively unimportant, so use the latest one you can find that came from the aforementioned repositories.
rpm -ivh krb5-workstation*.rpm
Another package to add is the pam_krb5 package from the same repository. This is not a requirement but will add better integration and protections to keep system accounts we discussed before from using AD authentication. You need version 2.11 of pam_krb5 to get the benefit of this capability for those emergency use accounts. Add the pam_krb5 package by doing the following:
rpm -Uvh pam_krb5*.rpm
The next step is to configure the VMware ESX firewall to allow AD and its components to speak with the host.
esxcfg-firewall -e activeDirectorKerberos esxcfg-firewall -o 445,tcp,out,MicrosoftDS esxcfg-firewall -o 445,udp,out,MicrosoftDS esxcfg-firewall -o 389,tcp,out,LDAP esxcfg-firewall -o 464,udp,out,kpasswd esxcfg-firewall -o 464,tcp,out,kpasswd
Then enable AD authentication using the following, which will enable the VMWARELAB active directory domain using the domain controller dc.vmwarelab.com. It should be noted that you do not need to specify the domain controller if your domain can resolve within DNS.
esxcfg-auth --enablead --addomain=VMWARELAB --addc=dc.vmwarelab.com
Modify the /etc/pam.d/system-auth file so that it looks similar to the following. Note that only the highlighted lines need to be modified. One is to add in the pam_access line we discussed previously, and the other is to ensure that the emergency use users are not authenticated using AD, which could be broken during an emergency.
#%PAM-1.0 # Autogenerated by esxcfg-auth account required /lib/security/$ISA/pam_unix.so broken_shadow account required /lib/security/$ISA/pam_krb5.so account [default=bad success=ok user_unknown=ignore] /lib/security/$ISA/pam_access.so auth required /lib/security/$ISA/pam_env.so auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok auth sufficient /lib/security/$ISA/pam_krb5.so use_first_pass minimum_uid=1000 auth required /lib/security/$ISA/pam_deny.so password required /lib/security/$ISA/pam_cracklib.so retry=3 password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow password sufficient /lib/security/$ISA/pam_krb5.so use_authtok password required /lib/security/$ISA/pam_deny.so session required /lib/security/$ISA/pam_limits.so session required /lib/security/$ISA/pam_unix.so session optional /lib/security/$ISA/pam_krb5.so
Now it is time to use the previously installed RPM programs to test the Kerberos connection that composes part of AD.
/usr/kerberos/bin/kinit Administrator Password for Administrator@VMWARELAB.COM: kinit(v5): Clock skew too great while getting initial credentials
If any errors occur like the one in the example, they need to be fixed. The one listed implies that the VMware ESX host is out of time sync with the domain controller. To fix properly, configure NTP on your VMware ESX host to match that used by your domain controller. Other errors require changing the encryption parameters used to establish the connection. To fix an encryption issue, edit the file /etc/pam.d/krb5.conf to look like the following with the appropriate changes for your domain. To fix an encryption problem we added the default_tkt_enctypes and default_tgs_enctypes lines to the existing file.
# Autogenerated by esxcfg-auth
[domain_realm]
vmwarelab = VMWARELAB
.vmwarelab = VMWARELAB
[libdefaults]
default_realm = VMWARELAB
default_tkt_enctypes = des3-hmac-sha1 des-cbc-crc des-cbc-md5 rc4-hmac
default_tgs_enctypes = des3-hmac-sha1 des-cbc-crc des-cbc-md5 rc4-hmac
[realms]
VMWARELAB = {
admin_server = dc.vmwarelab.com:464
default_domain = dc.vmwarelab.com
kdc = dc.vmwarelab.com:88
}
After kinit works, you have AD authentication available to you, and you just need to create and maintain local user accounts. AD uses LDAP to retrieve group and user information but uses Kerberos to retrieve authentication information. Also, note that the krb5-workstation package can now be removed because it is no longer needed, unless you want to keep it around to help solve AD integration problems. Remove using the following command:
rpm -e krb5-workstation
Partial Integration with LDAP over SSL or Secure LDAP
To use secure LDAP or LDAP over SSL you must follow the preceding steps for partial integration with AD. After you have that working, you need to modify the configuration to use secure LDAP. However, to do this we must first add another RPM package to the installation. This is the cyrus-sasl-gssapi package, and you can retrieve this from the same location you retrieved krb5-workstation. Install using the following line. Unlike the krb5-workstation package, you will not be able to remove this RPM when the integration is completed. If your company has concerns about third-party packages, this is not necessarily the integration you desire.
rpm -ivh cyrus-sasl-gssapi*rpm
Run the following command to add LDAP authentication to your existing AD integration.
esxcfg-auth --enableldapauth --ldapserver=vmwarelab.com --ldapbasedn=DC=vmwarelab, DC=com
Next edit the /etc/openldap/ldap.conf file to look like the following using your base DN and LDAP server as appropriate. Even though we specified them in the preceding command, we should also double-check everything.
BASE dc=vmwarelab,dc=com URI ldaps://vmwarelab.com:636 ldaps://vmwarelab.com:636 TLS_CACERT /etc/openldap/cacert.cer SASL_SECPROPS maxssf=0
The last line is required when using secure LDAP with Kerberos, which is the configuration we are using. The certificate specified, cacert.cer, points to a file containing your exported root certificate and your consolidated certificates. This you would get from your certificate authority. If DNS is configured properly and you have multiple LDAP servers, DNS will handle which server to query. Note that you will want to ensure your DNS is configured properly and all names are resolvable going forward.
Like the previous section, we need to further configure the firewall to allow SSL-based LDAP queries to be made to the LDAP server.
esxcfg-firewall -o 636,tcp,out,LDAP over SSL
Export your root certificates from your CA in a base64 encoded X.509 file. If a certificate chain exists, ensure they are placed in one file. Place this file in the location specified using the TLS_CACERT variable within the /etc/openldap/ldap.conf file. On the LDAP server, create an account to which you will bind and use to search the directory from your VMware ESX host.
Using kinit as we did within the "Partial Integration with AD" section, we test the integration once more and fix any issues that show up.
/usr/kerberos/bin/kinit Administrator Password for Administrator@VMWARELAB.COM: kinit(v5): Clock skew too great while getting initial credentials
Last, test to be sure that the SSL connection works as expected.
openssl s_client -CAfile /etc/openldap/cacert.cer -connect vmwarelab.com:636
If the last line reads "Verify return code: 0 (ok)", everything is set up properly and you can properly use LDAP over SSL. You now need to create and maintain local user accounts. You will authenticate using Kerberos and use LDAP over SSL to retrieve group and user information. Also, note that the krb5-workstation package can now be removed because it is no longer needed, unless you would like to use it to solve integration problems. One way to create and maintain local user accounts is to query the LDAP or LDAP over SSL server periodically using the following script from your VMware ESX service console. This script is reprinted here with permission from its author, Steve Beaver.
########################################################################
#!/bin/bash
# Secure LDAP Search Script to add and remove users based on Group
Membership
# Stephen Beaver
########################################################################
# variables
base="-b DC=domain,DC=com" # Replace with your domain name
# This is the user that we will bind to LDAP with
user="-D MyLDAPUser@domain,com" # or can be in the form of
# user="-D CN=MyLDAPUser, OU=OU, DC=Domain, DC=COM
pass="-w password" # The LDAP user password
group="ESX_ADMIN" # The directory group you will search for
esxgroup="ESX-Admin" # The ESX group you would like the users to be a
member off
programdir="/usr/LDAP" # The directory this script will run
from
# More Variables that do not need to be edited
cmd="ldapsearch -Y GSSAPI -LLL"
pipe="-u -tt -T ${programdir}"
pipe2="-u -tt -T ${programdir}/Member"
filter2="CN=${Group} member"
filtersam="samAccountName"
####################################################################
# Get Kerberos Ticket
echo password | /usr/kerberos/bin/kinit -V $user
# Sanity Check to make sure all the files and folders needed are in
place or create them
if test ! -x "$programdir" ; then
mkdir $programdir
mkdir $programdir/Member
mkdir $programdir/Member/New
mkdir $programdir/Member/Old
echo > $programdir/Member/New/$Group.txt
echo > $programdir/Member/Old/$Group.txt
fi
################ NEW SEARCH ################
# The first search to find the group and see who if any are members
LDAP_search ()
{
${cmd} ${base} ${user} ${pass} ${pipe} ${filter1}
if [ "$?" -ne "0" ]; then
printf "ERROR running LDAP Search script exiting"
return
fi
LDAP_search_member
}
# Now that I have a temp file for each user. I need to collect and list
in a file to read from
# If I find no users in the group then no need to continue. Return and
move on
LDAP_search_member ()
{
cd $programdir
ls -1 $programdir/ldapsearch-member-* > $programdir/filelist.txt
if [ "$?" -ne "0" ]; then
printf "No Members moving on... "
return
fi
declare LINE
declare MEMBER
cat $programdir/filelist.txt |
while read abc
do case $abc in
Member) echo $abc ;;
*) awk '{print $0}' $abc >> $programdir/ulist.txt ;;
esac
done
sed 's/,OU=.*//g' $programdir/ulist.txt > $programdir/mlist.txt
sed 's/CN=//g' $programdir/mlist.txt >
$programdir/Member/filelist.txt
LDAP_search_sam
}
# Now I have a list in a usable format. Time to search again to get the
samAccountName # or userid of each user in the group.
LDAP_search_sam ()
{
rm -R $programdir/ldapsearch*
rm -R $programdir/filelist.txt
rm -R $programdir/ulist.txt
rm -R $programdir/mlist.txt
mv -f $programdir/Member/New/$Group.txt
$programdir/Member/Old/$Group.txt
LDAP_search_create
}
# Now that I have a temp file for each user. I need to collect and list
in a file to read from
# Sort the list and compare the old with the new to see if I need to add
or remove users
# The useradd command below to add the user
LDAP_search_create ()
{
cd $programdir/Member
awk '{print $0}' $programdir/Member/filelist.txt | tr [:upper:]
[:lower:] >> $programdir/Member/$Group.txt
rm -R $programdir/Member/filelist.txt
mv -f $programdir/Member/$Group.txt
$programdir/Member/New/$Group.txt
sort -f -o $programdir/Member/New/$Group.txt
$programdir/Member/New/$Group.txt
comm -1 -3 $programdir/Member/New/$Group.txt
$programdir/Member/Old/$Group.txt > $programdir/remuser.txt
comm -2 -3 $programdir/Member/New/$Group.txt
$programdir/Member/Old/$Group.txt > $programdir/adduser.txt
cat $programdir/remuser.txt |
while read oldlist
do userdel -r $oldlist
done
rm -R $programdir/remuser.txt
cat $programdir/adduser.txt |
while read newlist
do useradd -M -g ESX-Admin $newlist
/usr/bin/chage -M 99999 $newlist
done
rm -R $programdir/adduser.txt
}
######### This section is the main body which calls all the functions
listed above
LDAP_search
exit
Full Integration with AD
Full integration with AD implies that you manage all the users directly from your AD server. In other words, there will be no need to create local accounts for administrators to be able to access the VMware ESX host. However, without pam_access implemented, as discussed previously, this is not a secure option, because AD users could log in to your VMware ESX service console if they have the network access to the box. So it is very important to configure pam_access properly.
Full integration starts at the end of the "Partial Integration with AD" section discussed previously, but instead of creating users, we will add functionality so that the users will no longer need to be created.
The first thing we do is modify /etc/pam.d/system-auth once more to change the following line:
account required /lib/security/$ISA/pam_krb5.so
Create this new line, where the default required keyword has been modified to look like the following line.
account [default=bad success=ok user_unknown=ignore] /lib/security/$ISA/pam_krb5.so
After that is completed, add one more line to the end of /etc/pam.d/system-auth to allow home directories to be created when users log in. This removes the need for extra management.
session required /lib/security/$ISA/pam_mkhomedir.so skel=/etc/skelumask=0077
Now we need to add a few more RPM packages from the ESX media: namely the samba-server package. We will not be running the entire Samba server but only a small part of it, because enabling the Samba server opens up the ESX host for possible SMB/CIFS attacks.
rpm -ivh samba-server*rpm
In general, if you are allowed to do so you will want to update all Samba packages to a minimum of v3.0.25 to alleviate the need to make any changes to your AD server to lower its security stance as the lack of encryption could lead to a MiTM vulnerability. The changes you may have to make are to the AD servers' local security policies to disable the following options.
Domain member: Digitally encrypt or sign secure channel data (always) Microsoft network server: Digitally sign communications (always)
Next modify /etc/samba/smb.conf to look like the following so that the winbind daemon can be used to query authorization and credential information from the AD server.
[global]
workgroup = VMWARELAB
server string = Samba Server
printcap name = /etc/printcap
load printers = no
cups options = raw
log file = /var/log/samba/%m.log
max log size = 50
security = ads
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
dns proxy = no
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
template shell = /bin/bash
template homedir = /home/%D/%U
winbind use default domain = yes
password server = dc.vmwarelab.com
realm = VMWARELAB.COM
The template_homedir option will be used in conjunction with the pam_mkhomedir module added previously to the end of the /etc/pam.d/system-auth file. To complete this, we need to create a directory and set up permissions to allow the creation of the user directories and assigning the proper permissions automatically. If user directories are not set up, the user will be placed within the top level directory on log in. This is an undesirable result. The template_homedir option in /etc/samba/smb.conf contains two variables, %D, which refers to the domain to use, in our example VMWARELAB, and the %U variable, which refers to the user directory to create. The following commands create the directory for the domain and set the permissions of the directory so that when users create files within it, they assume the ownership of the user creating them and not the root user.
mkdir /home/VMWARELAB chmod 1777 /home/VMWARELAB mkdir /var/log/samba
Before we join the VMware ESX host to the domain, we need to modify the /etc/nsswitch.conf file so that queries for users and groups go through winbind instead of just querying the local files. Add the winbind keyword to the highlighted lines per the following example.
# Autogenerated by esxcfg-auth aliases: files nisplus automount: files nisplus bootparams: nisplus [NOTFOUND=return] files ethers: files group: files winbind hosts: files dns netgroup: nisplus netmasks: files networks: files passwd: files winbind protocols: files publickey: nisplus rpc: files services: files shadow: files winbind
Now we are ready to join the VMware ESX host to the domain. If the kinit test outlined in the previous section "Partial Integration with AD" passed, it is safe to add the host to the domain using the following commands. They will not only join the host to the domain but start winbind and enable it to start on reboot of the host.
net ads join -UAdministrator Administrator's password: Using short domain name -- VMWARELAB Joined 'HOST' to realm 'VMWARELAB.COM' service winbind start chkconfg winbind on
Winbind is an important aspect of this type of integration because it will speak to the AD server in an encrypted fashion and is the only part of the Samba server package that we will be using. At no point should the smb daemon be started or need to be started. Now it is time to test our configuration using a winbind tool named wbinfo.
Verify that the groups are picked up from the AD server.
wbinfo -g domain computers domain controllers schema admins enterprise admins domain admins domain users domain guests group policy creator owners
Verify that users are picked up from the AD server.
wbinfo -u administrator guest support_388945a0 krbtgt testauser smbservice
Verify that trusted secret via RPC calls succeed. Note and fix any errors.
wbinfo -t
Verify that VMware ESX sees the AD groups properly.
getent group root:x:0:root ... domain computers:*:16777220: domain controllers:*:16777218: schema admins:*:16777222:administrator enterprise admins:*:16777223:administrator domain admins:*:16777219:administrator domain users:*:16777216: domain guests:*:16777217: group policy creator owners:*:16777224:administrator
Verify that VMware ESX can resolve the AD users. Note that this could list hosts as well as users, depending on how the organizational unit was set up within AD. Note that the part of the command return should be the path we set up to be used by the pam_mkhomedir module.
getent passwd root:x:0:0:root:/root:/bin/bash ... administrator:*:16777216:16777216:Administrator:/home/VMWARELAB/administ rator:/bin/bash guest:*:16777217:16777217:Guest:/home/VMWARELAB/guest:/bin/bash ... krbtgt:*:16777220:16777216:krbtgt:/home/VMWARELAB/krbtgt:/bin/bash
Verify that an AD user picks up the proper user ID and group ID for a specific user as well as the complete group list associated with the user. Compare the results to the results found in the previous test. It should also be noted that sometimes AD integration has issues if a user belongs to too many groups. With the latest versions of Samba, this should no longer be the case.
id testauser uid=16777221(testauser) gid=16777216(domain users) groups=16777216(domain users)
Now we have full integration with AD, which does not require user management on all your ESX hosts. Occasionally, you will want to go through and remove the deleted users from the /home/%D directory we created previously. This could easily be accomplished with a simple script to query the AD server and remove any directories for the users that do not exist within the directory service. This script follows and could be run from within the cron daemon at least once per day.
wbinfo -u > /tmp/wbinfo.$$
for x in `ls /home/VMWARELAB`
do
u=`basename $x`
grep "^$u$" /tmp/wbinfo.$$ >& /dev/null
if [ $? -gt 0]
then
rm -rf /home/VMWARELAB/$u
fi
done
rm -rf /tmp/wbinfo.$$
If there are reasons you would not use winbind, you can also configure LDAP or SSL over LDAP to query the same information using different tools but the same approach.
Setting Up Directory Services on Other Management Hosts
It is also very important to set up directory services on all the workstations and hosts in use by other aspects of the virtual environment. This includes but is not limited to backup servers, those workstations running the virtual machine management tools, the Virtual Infrastructure Management Appliance (VIMA) from VMware if it is in use, hosts for LifeCycle, Lab, Site, and Update Managers, as well as any hosts used for monitoring the virtual environment.
This is where those well-defined user roles come into play; you must maintain the same view of all data across a multitude of hosts, virtual machines, and possibly appliances.
Lifecycle manager will use directory services within itself, yet Lab and Site manager tools will pick up roles and permissions from vCenter. Update manager uses different authentication as well.
Setting up directory services on these hosts is outside the scope of the book, but nonetheless should be done to maintain auditing across the entire management spectrum on a remote logging host.