Much has been done in the advancement of IT infrastructure management technologies, but there is no fundamental change in the way the operating systems are installed on the bare-metal Industry Standard Servers. There are scores of tools available that help the IT admins to automate the OS installation. Almost all bare-metal OS deployment tools rely on outdated technologies and protocols such as PXE, TFTP, IPMI, DHCP, legacy BIOS because they are supported by multiple vendor servers as well as multiple generation servers. However, these underlying technologies and protocols have limitations and security vulnerabilities that go against the security policies that are key to safeguarding modern datacenters against cybersecurity threats. These legacy technologies also increase the complexity of the IT operations. They are also not designed to handle the deployment of hundreds of servers, which in turn impacts how fast an enterprise can respond to market demands.
(Pre-Boot Execution Environment) was introduced as part of the Wired for Management framework in the 1990s. PXE allows a server to boot from local network and, the IT administrator to install and configure the OS on networked computers remotely. However, one of the key issues with PXE is the lack of security. It has no mechanism for encryption or authentication, making it susceptible to man-in-the-middle attacks. PXE requires servers with PXE-capable NIC adapters and relies on TCP/UDP, DHCP and TFTP. The following sections describe the various underlying technologies involved with PXE based OS deployment tools:
TFTP
PXE uses TFTP service for presenting the boot loader and the OS installation media to the servers over the network. TFTP expands to Trivial File Transfer Protocol. It is basically FTP without login or access control mechanisms. TFTP has reliability issues associated with time-outs and UDP packet loss. It is also not designed to handle larger file sizes of boot images that come with today's operating systems. The TFTP and UDP transactions associated with PXE may be the last unencrypted traffic on your network and are trivial to intercept. This boot process goes against the “zero trust” concept applied to today’s networks (Read more).
IPMI
IPMI is a legacy out-of-band systems management protocol to communicate with server BMC for management and monitoring capabilities. It runs on TCP/UDP protocol. The legacy OS deployment tools depend on IPMI for controlling server power ON/OFF states during the installation process, and modifying BIOS settings. IPMI exposes serious security holes that allow the attackers to gain control on the server, bypassing the security restrictions imposed in the host operating system. When certain IPMI options are selected, they can allow access to the server bypassing user authentication.
IPMI tools based on IPMI 2.0 specification can reveal critical information through session data even when the wrong password is provided. This critical information can then be used offline to determine the actual password. Once the attacker gains access to IPMI, the attacker can power ON/OFF the server, change BIOS settings, network boot the server and so on. As IPMI operates in OOB (Out Of Band management network), the host operating system security defenses will not be able to prevent such attacks.
The DMTF organization has developed a secure and scalable interface specification called Redfish to work in modern datacenter environments. Redfish uses HTTPS encrypted connectivity for a secure and reliable computer interface for server management. All vendor BMC's come with IPMI support. However, it is a good practice to disable IPMI(over LAN UDP port) on BMC and solely rely on Redfish. Most latest generation server BMCs by default are factory-configured to disable the IPMI over LAN UDP port.
DHCP
PXE requires a DHCP service for establishing a network connection with target servers, and also for configuring host network settings.
However, DHCP poses security risks. The DHCP protocol requires no authentication, allowing any client to quickly join a network. Because of this limitation, it opens up many security risks, including DHCP clients receiving bad information from unauthorized DHCP servers and unauthorized/malicious DHCP clients receiving IP addresses. A rouge DHCP server can give incorrect network information to cause denial-of-service attacks or man-in-the-middle attacks. There are some security measures available to mitigate these risks. However, eliminating DHCP itself is the best solution it is feasible. DHCP can be replaced with static IP assignment or IP pools that can provide the next available IP address for new hosts. But, PXE cannot support these alternatives.
BIOS (Basic Input Output System-based)
BIOS originated in the early development of personal computers back in 1980s. The BIOS loads when the server starts up. It is responsible for waking up the hardware components, as well as testing and initializing them. The BIOS then looks for a Master Boot Record or MBR stored on the boot device for booting the operating system.
However, there are several limitations with legacy BIOS. It supports only 16-bit mode with 1MB memory address space, limiting its ability to handle multiple hardware devices at once. This limitation makes the boot process slower. Also, the maximum boot drive size is limited to only 2.1TB. 3 TB drives are now common, and hence a server with a BIOS can’t boot from them.
Almost all current generation industry standard servers come with support for UEFI (Unified Extensible Firmware Interface) which is the new standard for server boot technologies. UEFI has several advantages over legacy BIOS. UEFI technology was originally developed several years ago for the non-x86 HP-Intel Itanium server architecture. It is also the technology used on Macintosh computers. Eventually it was transferred to the Unified EFI Forum, which governs the specification today. Key benefits of UEFI are:
- UEFI is faster to boot.
- It functions in both 32-bit and 64-bit modes with much greater memory address space.
- With SecureBoot enabled it can authenticate firmware and OS using encrypted digital certificate to prevent boot time malware attacks.
- UEFI uses GPT (GUID Partition Table) partition-table which has the theoretically upper limit of 9.4 ZB (zettabytes) (ref.).
- UEFI is designed to be architecture-independent and supports any CPU architecture.
- UEFI is capable of network operations with built-in network stack. However, there is only support for HTTP but not for HTTPS.
- UEFI conforms to the Redfish API standard.
While latest generation servers come with both Legacy BIOS and UEFI boot options, the factory settings enable UEFI for default boot mode. HPE Gen9 and later generations have UEFI set as the default boot mode.
Alternatives to PXE
iPXE
iPXE is a sophisticated boot program with advanced features such as HTTPS, Digest authentication, POST requests, scripts, menus, customizable code signing etc. iPXE also offers boot from a variety of sources such as HTTP, iSCSI SAN, FCoE, WAN etc. iPXE works as the initial bootloader with an iPXE configuration file that selects the next bootloader based on the MAC address or other criteria. Though there are different ways to let servers boot into iPXE, most implementations use the same old PXE/TFTP/DHCP to boot iPXE first, and then use it to chain load the actual OS installation media. iPXE feature is included in UEFI firmware for some servers and for such servers there is no need for an additional TFTP server. Setting up iPXE environment is not simple as it involves setting up DHCP and TFTP (in certain scenarios) first.
UEFI-HTTP Boot
UEFI-HTTP boot is a UEFI feature for net-booting which is similar to PXE and uses HTTP instead of TFTP. UEFI HTTP Boot can be used to chainload iPXE from a HTTP server, eliminating the need for a separate TFTP server. This is helpful when iPXE support is not part of the server UEFI firmware. However, it relies on DHCP for IP configuration. Also, UEFI network stack only supports HTTP and not HTTPS. So, there is a dependency on HTTP net-booting which is not a problem in most datacenters. But in high security environments where HTTP is not allowed, this dependency can be a problem.
Virtual Media
Virtual Media is a feature of server BMC that allows attaching URL-based media as virtual CD/DVD-ROM or USB device. This feature can be used to boot server from the network using bootable image from HTTP or HTTPS file servers. Virtual Media is different from TFTP because the boot media is accessed through a more secure and reliable HTTP(s) protocol and is relatively easy to setup. The virtual media feature supports ISO images of up to 8 TB, and can perform better with higher network bandwidths. One key advantage that Virtual Media has over PXE/iPXE and UEFI-HTTP is that it doesn't need to utilize the server network ports to access the media over network. It uses out-of-band networking to access the boot and installation media, as it uses the server BMC network port. And since this port is commonly connected to an isolated management network, there is no interference to the production network by Virtual Media.
Virtual Media can be a good alternative to PXE. It just needs bootable OS installation media with embedded kickstart to automate OS installation and configuration. The kickstart file can be coded to use DHCP or Static IP for configuring the host networking.
RedefinIT BMA - OS deployment tool for modern datacenters
RedefinIT BMA (Bare Metal Automation) reduces the complexity in bare-metal server deployment by providing a simplified approach to automate the OS installation and Firmware update process. It uses modern industry standard technologies such as DMTF Redfish, Virtual Media, UEFI, HTTPS/TLS for faster, reliable and secure bare-metal server provisioning. This product supports multi-vendor server hardware, operating systems and hypervisors making it a versatile tool in the modern enterprise datacenter. BMA provides RESTful API interface to easily integrate with O&M (Orchestration and Management) tools such as Ansible, Terraform etc. for complete end-to-end automation of Day-1 and Day-2 operations for bare-metal server infrastructure.
Benefits:
- Offers Rich Web GUI and RESTful API secured by HTTPS/TLS.
- Uses Redfish for server management and hardware discovery.
- Can combine firmware update and OS installation into single operation to onboard servers within minutes.
- The tool is easy to install and requires minimum preparation work before the deployments.
- Uses Virtual Media for netbooting through OOB management network. There is no requirement for plumbing additional PXE network.
- Current version supports HPE Gen10 and Dell Gen14 rackmount servers.
- Accepts static IP address for host network configuration. Accepts IP pool or a range of IP addresses for bulk deployment mode. DHCP is also supported.