Organizations must develop and maintain documented policies for security patching that extend to all their platforms

All significant IT governance frameworks require policy documentation for changes in the production environment (including security patches). The following guidelines offer best practices for change management:

• Information Technology Infrastructure Library (ITIL)

• Control Objectives for Information and Related Technology (COBIT)

• Information technology — Security techniques — Code of practice for information security management ISO/IEC 27009 (formerly known as ISO17799)

The primary objectives of documented policies is to overcome the fear associated with introducing changes in production environment and providing organizations with a repeatable process when applying required changes.

Organizational policies must clearly state objective criteria to guide patching and upgrade decisions as well as identify the complete approval process for these decisions

An important objective of an organization’s documented policies should be to clearly state under which conditions and when security patches are applied to critical systems. In the recent IOUG and Oracle survey on security assurance, a large proportion of respondents indicated that their organizations’ policies (if they existed) required them to justify the application of security patches. Such justification should be based on an organization’s risk tolerance for each of its systems.

For example, organizational policies may require timely application of security patches for critical systems, but noncritical systems could be delayed if other mitigation measures are available. However, such an approach should always be validated by the IT security group to avoid that the compromise of a noncritical server leads to compromising more critical servers, as a result of the chain of trust existing between servers. In addition, auditors should vet this policy to ensure that it meets their compliance requirements.

Exceptions to patching decisions must be documented, and accountability for noncompliance should be clearly established

Even when documented policies exist, an absence of accountability may result in security patches not being applied. In organizations with high security requirements, timely application of all critical security patches is typically required, and any deviation is documented along with the identity of the person or organization that authorized the policy deviation along with the appropriate justification. The goal of such an approach is to ensure that the benefits of non-compliance exceed the security risk assumed by the organization. As much as possible, the decision to skip patches and increase the organization’s risk exposure should lie with the affected business unit. However, IT, legal, and other departments must help assess the risk exposure from their individual perspectives.

IT must invest in the appropriate tools and document repeatable processes for effective patching

With the right tools, the time needed to adequately test and apply patches is a small fraction of the time needed to perform the same tasks manually. Furthermore, organizational policies should create an environment that fosters a proper level of security for all systems. Organizations should follow the following best practices:

• Document and regularly review the organization’s service level objectives and agreements.

Security measures should be incorporated into an organization’s service level agreements. This means that specific maintenance time for testing and deploying security patches should be provided in the organizational policies.

• Keep track of system configurations and understand their interdependencies.

A good configuration management solution should keep track of and dynamically detect changes to IT assets. The solution should automatically collect key configuration information about servers, databases, middleware, applications, and other components such as point-of-sale devices. Good configuration management solutions also provide users with templates and blueprints for managing complex application environments such as enterprise resource planning, customer relationship management, and custom-built applications. These templates are critical to helping the IT staff understand the intricacies of the IT environment and make quick decisions about where, when, and how to apply security patches.

• Utilize a formal change management process.

Unauthorized or unintended changes can be the single most important cause of vulnerability and system downtime. Documenting changes with a help desk or change management system is useful not only for compliance purposes, it also provides essential information about the circumstances surrounding a patch request. Later, such information can help an organization quickly determine why a patch was applied, who requested it, and what business requirements triggered the request. Integration with the back-end systems management software is another important requirement, because although a patch request may be entered by service-department personnel, it may be subject to formal approval by a number of different persons, including non-IT personnel, and it is usually applied by a systems or database administrator. Closed-loop integration between the help desk/change management system and the back-end systems management tool allows automatic verification of patch application.

• Develop standard systems configurations and security baselines.

It is often hard to tell how many changes have been made to a particular system even when formal records exist. It is a good practice to maintain a library of standard images or configurations. When changes are required, it is best to update the standard configurations first. After proper testing, these changes should be rolled out to the organizations using applications provisioning and patch automation tools to maintain consistency across all systems configurations.

• Invest in a good testing tool.

Inevitably, changes must be tested to prevent unforeseen consequences such as system downtime or denial of service. Automated load and performance testing can scale testing efforts to provide an accurate representation of the production environment. Organizations should be aware of code coverage issues, because some tools require manually generated test scripts. But these scripts are time-consuming to create, so testers may limit the scope of testing and miss critical areas. Organizations should choose tools that automatically generate scripts and record and replay actual production workloads whenever possible.

• Mask your test data…or else!

Proper testing requires an accurate depiction of enterprise data. Organizations should always mask production data at the source to avoid unauthorized access in the testing environment, which often has looser access control policies than production systems.

• Maintain proper production monitoring.

Administrators must remain vigilant about the potential effect on production users when newly patched systems return to production. Good monitoring tools specifically designed for IT administrators do not require instrumentation or deep knowledge of the application code, and can be safely deployed on production systems. When choosing a tool, consider how much overhead it adds to a production system and whether it can detect changes and update its own model of the application. Model-driven diagnostics and application performance management (APM) is becoming increasingly popular, and if your applications are based on a service-oriented architecture, model-driven APM is your best bet.

A vendor’s security assurance practices should be reviewed before acquiring new products.

A vendor’s patching procedures are very visible to customers —the cost of security patches greatly contributes to customers’ security management expense. It is critical that customers understand their vendor’s patching procedures, especially regarding predictability (when are patches released?), availability (are patches available on all supported versions and platforms?), and transparency (What kind of vulnerability and patch documentation is provided by the vendor?). Ad hoc patch releases are very costly to customers because the lack of predicatbility makes it very difficult to adopt a repeatable patching process. In addition, there are other aspects to a vendor’s security assurance practices that are relevant to customers—a vendor’s level of sophistication in preventing security vulnerabilities and ensuring the effectiveness of the security features in its software, for example.

Applying security patches in a timely manner is a critical aspect of sound IT governance. Appropriate patching practices help an organization maintain an optimal security posture. Although it may not always be possible to ensure that all security patches are applied systematically, organizations should try to foster an environment in which IT administrators have the right tools and incentives to maintain optimal system security. Above all, organizations should implement the proper policies to maintain and care for critical systems, and to help overcome the fear associated with altering these critical systems.

About the authors: Eric Maurice is a Director for Software Security Assurance at Oracle. Moe Fardoost is a senior director of product marketing at Oracle focusing on IT operations challenges.