In recent years, the popularity of fast internet connections that are virtually always on has enabled software vendors to embed automated update mechanisms into many desktop programs. Though some users may find these updates cumbersome, most have come to accept the daily routine of updating their antivirus software and periodically installing critical Windows updates.
However, the generalization of automated software updates may ultimately create security issues for organizations. Who knows what updates are being pushed to user desktops and how they are actually installed? Who is responsible for preventing the installation of malware disguised as security tools?
Although the practice of frequent client software updates and patching is generally accepted, particularly when these updates are security related; organizations and IT administrators are often reluctant to deploy patches on production servers regardless of the frequency of their release by vendors.
In this article, we explore how the psychology and work environments of IT administrators can play a significant role in preventing the timely deployment of security updates to production servers. We also examine why users and organizations so rarely apply security patches onto servers and systems in a timely fashion. Finally, we provide recommendations to help IT staff deal with the challenges associated with patching production servers.
Administrators generally cite extremely tight maintenance windows as the main reason why security patches are not consistently applied onto production servers. In other words, administrators believe that their service level agreements (internal or external) do not allow enough time to bring down systems and apply the necessary updates and security patches.
However, even when vendors do provide a predictable patching schedule, many organizations still do not apply these patches in a timely fashion. Perhaps the reluctance to apply patches or introduce changes in a production environment is attributable to more than just tight maintenance windows?
The resistance of organizations and IT administrators to apply patches may have less to do with maintenance windows and more to do with the cost and ressources required for adequately testing changes to prevent unforeseen outages. Production environments generally perform in a highly predictable fashion, and organizations and administrators are under pressure to ensure that these systems continue to do so.
In a recent survey conducted jointly by the Independent Oracle Users Group (IOUG) and Oracle, respondents cited more time (wider maintenance windows) as the least compelling reason that would cause them to apply security updates more quickly or consistently. According to survey results, stronger motivators for applying security patches included better tools and documentation for testing and deployment, executive mandate within the organization, occurrence of a massive malware outbreak or a failed security audit. It is important to note that these responses were solicited from those responsible for the testing, deployment, or approval of security updates in their organizations production systems.
The survey provided ample opportunities for participants to include comments. One common feedback was that organizational policies for security patching are typically limited to the desktop environment. Respondents felt that security flaws addressed in security patches in production servers are generally mitigated by security measures external to the affected systems.
The most telling aspect of this survey was that respondents often expressed anxiety or even fear about altering production systems. These business-critical systems must operate in a predictable fashion, and are considered too complex to tinker with. The combination of these factors fosters a situation in which organizations are not likely to apply security patches. This creates a paradox: The importance of the systems and the expectation of their near-always availability are obstacles to properly maintaining and securing business-critical systems.
Organizations must find a way to stay reasonably current with security patches while meeting their service level agreements. The following recommendations can help organizations more efficiently tackle security patching.
Organizations must develop and maintain documented policies for security patching that extend to all their platforms
All significant IT governance frameworks require policy documentation for changes in the production environment (including security patches). The following guidelines offer best practices for change management:
Information Technology Infrastructure Library (ITIL)
Control Objectives for Information and Related Technology (COBIT)
Information technology Security techniques Code of practice for information security management ISO/IEC 27009 (formerly known as ISO17799)
The primary objectives of documented policies is to overcome the fear associated with introducing changes in production environment and providing organizations with a repeatable process when applying required changes.
Organizational policies must clearly state objective criteria to guide patching and upgrade decisions as well as identify the complete approval process for these decisions
An important objective of an organizations documented policies should be to clearly state under which conditions and when security patches are applied to critical systems. In the recent IOUG and Oracle survey on security assurance, a large proportion of respondents indicated that their organizations policies (if they existed) required them to justify the application of security patches. Such justification should be based on an organizations risk tolerance for each of its systems.
For example, organizational policies may require timely application of security patches for critical systems, but noncritical systems could be delayed if other mitigation measures are available. However, such an approach should always be validated by the IT security group to avoid that the compromise of a noncritical server leads to compromising more critical servers, as a result of the chain of trust existing between servers. In addition, auditors should vet this policy to ensure that it meets their compliance requirements.
Exceptions to patching decisions must be documented, and accountability for noncompliance should be clearly established
Even when documented policies exist, an absence of accountability may result in security patches not being applied. In organizations with high security requirements, timely application of all critical security patches is typically required, and any deviation is documented along with the identity of the person or organization that authorized the policy deviation along with the appropriate justification. The goal of such an approach is to ensure that the benefits of non-compliance exceed the security risk assumed by the organization. As much as possible, the decision to skip patches and increase the organizations risk exposure should lie with the affected business unit. However, IT, legal, and other departments must help assess the risk exposure from their individual perspectives.
IT must invest in the appropriate tools and document repeatable processes for effective patching
With the right tools, the time needed to adequately test and apply patches is a small fraction of the time needed to perform the same tasks manually. Furthermore, organizational policies should create an environment that fosters a proper level of security for all systems. Organizations should follow the following best practices:
Document and regularly review the organizations service level objectives and agreements.Security measures should be incorporated into an organizations service level agreements. This means that specific maintenance time for testing and deploying security patches should be provided in the organizational policies.
Keep track of system configurations and understand their interdependencies.
A good configuration management solution should keep track of and dynamically detect changes to IT assets. The solution should automatically collect key configuration information about servers, databases, middleware, applications, and other components such as point-of-sale devices. Good configuration management solutions also provide users with templates and blueprints for managing complex application environments such as enterprise resource planning, customer relationship management, and custom-built applications. These templates are critical to helping the IT staff understand the intricacies of the IT environment and make quick decisions about where, when, and how to apply security patches.
Utilize a formal change management process.
Unauthorized or unintended changes can be the single most important cause of vulnerability and system downtime. Documenting changes with a help desk or change management system is useful not only for compliance purposes, it also provides essential information about the circumstances surrounding a patch request. Later, such information can help an organization quickly determine why a patch was applied, who requested it, and what business requirements triggered the request. Integration with the back-end systems management software is another important requirement, because although a patch request may be entered by service-department personnel, it may be subject to formal approval by a number of different persons, including non-IT personnel, and it is usually applied by a systems or database administrator. Closed-loop integration between the help desk/change management system and the back-end systems management tool allows automatic verification of patch application.
Develop standard systems configurations and security baselines.
It is often hard to tell how many changes have been made to a particular system even when formal records exist. It is a good practice to maintain a library of standard images or configurations. When changes are required, it is best to update the standard configurations first. After proper testing, these changes should be rolled out to the organizations using applications provisioning and patch automation tools to maintain consistency across all systems configurations.
Invest in a good testing tool.
Inevitably, changes must be tested to prevent unforeseen consequences such as system downtime or denial of service. Automated load and performance testing can scale testing efforts to provide an accurate representation of the production environment. Organizations should be aware of code coverage issues, because some tools require manually generated test scripts. But these scripts are time-consuming to create, so testers may limit the scope of testing and miss critical areas. Organizations should choose tools that automatically generate scripts and record and replay actual production workloads whenever possible.
Mask your test data or else!
Proper testing requires an accurate depiction of enterprise data. Organizations should always mask production data at the source to avoid unauthorized access in the testing environment, which often has looser access control policies than production systems.
Maintain proper production monitoring.
Administrators must remain vigilant about the potential effect on production users when newly patched systems return to production. Good monitoring tools specifically designed for IT administrators do not require instrumentation or deep knowledge of the application code, and can be safely deployed on production systems. When choosing a tool, consider how much overhead it adds to a production system and whether it can detect changes and update its own model of the application. Model-driven diagnostics and application performance management (APM) is becoming increasingly popular, and if your applications are based on a service-oriented architecture, model-driven APM is your best bet.
A vendors security assurance practices should be reviewed before acquiring new products.
A vendors patching procedures are very visible to customers the cost of security patches greatly contributes to customers security management expense. It is critical that customers understand their vendors patching procedures, especially regarding predictability (when are patches released?), availability (are patches available on all supported versions and platforms?), and transparency (What kind of vulnerability and patch documentation is provided by the vendor?). Ad hoc patch releases are very costly to customers because the lack of predicatbility makes it very difficult to adopt a repeatable patching process. In addition, there are other aspects to a vendors security assurance practices that are relevant to customersa vendors level of sophistication in preventing security vulnerabilities and ensuring the effectiveness of the security features in its software, for example.
Applying security patches in a timely manner is a critical aspect of sound IT governance. Appropriate patching practices help an organization maintain an optimal security posture. Although it may not always be possible to ensure that all security patches are applied systematically, organizations should try to foster an environment in which IT administrators have the right tools and incentives to maintain optimal system security. Above all, organizations should implement the proper policies to maintain and care for critical systems, and to help overcome the fear associated with altering these critical systems.
About the authors: Eric Maurice is a Director for Software Security Assurance at Oracle. Moe Fardoost is a senior director of product marketing at Oracle focusing on IT operations challenges.