Everyone needs effective patch management. This critical but tedious process secures organizations of all sizes by eliminating vulnerabilities and delivering product upgrades.
Patching requires urgency. Attackers begin to reverse engineer patches immediately to exploit unpatched systems, even as organizations can leave known vulnerabilities unpatched for years. A Ponemon Institute survey of breach victims found that 60% of attacks exploited known vulnerabilities that had patches that hadn’t been applied yet.
Patching remains a problem because the details of execution can overwhelm an organization. Best practices for patching apply to organizations of all sizes and can improve time-to-patch, but the implementation will vary enormously depending on resources.
Best practices for patch management include:
- Asset Management: Know what you have and the current status.
- Risk Management: Know how important systems are and the risk to the organization if they fail or are compromised.
- Documentation: Set expectations, track status, and communicate clearly.
- Effective Patching Process: Effectively and efficiently obtain the correct patches, ensure their effectiveness, and deploy them with minimal operational impact.
- Mistake Management: Plans, processes, and technology can fail so know what to do to recover.
This article will explore each practice in detail to explain each role, minimum expectations, and provide tips to help make a smooth process. For a quick review of the topic, see: What is Patch Management?
1. Asset Management
To know what to patch, an organization first needs to know what it has. This is more formally called Asset Discovery, which the NIST Cybersecurity Framework, CIS Critical Security Controls, and other frameworks use as the foundation on which to build a strong cybersecurity program.
Minimum Expectations
Use automated tools. A clipboard and paper or excel inventory only works for the very smallest organizations. Even modest-sized organizations adopt automated tools to scour the network for every device, endpoint, server, operating system (OS), and application. Yet beware, some automated software may only track hardware, so a patch management team needs to understand tool limitations.
Track critical assets. If the failure of the asset can disrupt the business or cause costly data breaches, track it. Most organizations track all endpoint terminals (PCs, laptops, tablets, mobile phones, etc.), servers, cloud resources (containers, repositories, and other in-house managed infrastructure), critical software (accounting software, HR software, etc.), and network equipment used regularly.
Next Level Implementation
Track all assets, not just computers. Larger organizations understand the risk of ignoring the internet of things (security cameras, Wi-Fi enabled TVs, etc.), operational technology (wi-fi controlled pumps, temperature sensors, etc.), medical technology (especially wireless devices), point-of-sale (POS) systems, and rarely used assets.
Track app assets. Asset tracking also applies to applications developed in-house. While this will probably be managed by a separate DevSecOps team, the chief security officer should still receive reports regarding the software bill of materials (SBOM) and the status (version, last update, etc.) of application components.
Anticipate exceptions that may require special controls or planning. For example, in a hospital, the patient record database cannot go down and the Windows 95 machine controlling the MRI machine cannot be upgraded. These will need to fall outside of the standard processes for special handling or extra security controls.
Also, users need the ability to delay patches temporarily. No IT team wants to explain to the VP of Sales why mandatory update restarts disrupted a very important presentation. However, IT teams also need the ability to force restarts for employees that continuously push off updates.
Pro Tips
Effective asset management tools can often identify the current versions of the software or firmware. This will make it easier to track assets and determine if patches or more current versions may be available.
Standardized assets can reduce the burden on the IT team to track assets and their current status. Many organizations replace PCs in larger batches or limit software installations to reduce the number of hardware and software configurations to track and maintain. Organizations centralizing control or executing a merger will often inventory software categories (accounting, HR, etc.) and switch all users onto a single software package to help simplify updates (among other benefits).
Place assets in context using various network maps and diagrams to identify network segments, networking equipment, firewalls, and security controls that protect each asset. This context will play a role later in determining the overall risk of unapplied patches.
Quarantine unpatched unauthorized and BYOD devices that fall outside of the organization’s patch management control. Network access control (NAC) solutions can detect and quarantine devices that lack security patches and updates to protect the network.
Also see: Top IT Asset Management (ITAM) Tools for Security
2. Risk Management
Risk management requires an organization to understand the value of an asset (device, database, application, etc.) and the value of data contained in, secured by, or flowing through that asset. Risk should also reflect the damage to the organization if the asset fails or becomes compromised.
Minimum Expectations
Start with the most important. At the very least, risk management profiles should be in place for the most valuable and operationally critical infrastructure most likely to be targeted or affected by an attack. The number of assets that can be tracked may depend upon the resources of the organization and its ability to use the information effectively.
Characterize the likelihood of an attack based upon the security and operational controls defending the asset. Ratings can be categorical such as High/Medium/Low or rated 1/2/3/4 and these ratings should be used to prioritize the importance of patching the asset. For example, a data storage server rated medium value and low risk of attack will be of lower priority than the customer application server rated of high value and high risk receives its patches.
Next Level Implementation
Use risk management software to manage and maintain current risk profiles of tracked assets. The better software will enable an enterprise to characterize the risk in currency (US dollars, Euros, etc.) to help justify security spending to protect the asset. Ideally, the risk management, asset management, patch management and other related software can interconnect to share information and enable real-time analysis, automated prioritization, and alerts.
Group assets by risk class to help manage large numbers of assets, enable operational efficiency, and effective prioritization. A categorical asset class can speed up the assignment of a risk to an asset. However, this technique is best for large numbers of non-critical assets (workstations, switches, etc.). Key assets should be individually assessed.
Segregate patch-exempt assets. Not all assets can be patched. While upgrades may be preferred, sometimes it isn’t possible so the devices need to be segregated. For example, hardcoded credentials in IoT devices (especially medical technology) may require those devices to be placed on a separate network segment with a whitelist device list or similar security controls that protect against access from unauthorized access.
Pro Tips
Harden security by default. Many organizations allow vulnerabilities and open ports to remain on assets perceived to be buried within the network behind firewalls and other controls. However, zero trust philosophies require organizations to assume compromised networks and the frequent effectiveness of ransomware attacks indicates this assumption will be accurate more often than an organization might like.
Disabling unneeded protocols and unnecessary ports to harden assets will reduce a potential attack surface. Hardened infrastructure also lowers the overall risk of the organization – and reduces the urgency for some patches.
Context matters. Risk scores should incorporate the results of current vulnerability scans, possible public access (internet connections, physically accessible in a public place, etc.), and known vulnerabilities left unpatched.
Upgrade and retire legacy technology. Some technologies, especially in hospitals and for industrial applications, enjoy a lifespan that outlives the operating system required to manage the technology. MedTech and OT is notorious for requiring the maintenance of Windows 95, Windows 7, and other obsolete operating systems that can no longer be patched.
The cost to protect these systems continually increases both in hard dollars as well as the labor cost of the IT security team to implement controls. Organizations need to accurately assess these costs and invest in replacing obsolete technology.
Keep in mind that some upgrades may be possible that accomplish the same goals. For example, an obsolete Windows 95 computer can often be replaced with a fully updated Windows 11 machine that can host a secured virtual environment with a Windows 95 emulator.
Blocking potential attack paths for assets that can’t be patched or haven’t yet been patched is another defensive tactic, a process known as virtual patching.
Account for legal risk. When calculating the risk of an asset consider the legal liability for the organization if an adversary gains control. What would the legal liability cost the organization if the asset was unpatched and a judge, jury, or regulator found the organization to be negligent?
See the Best Patch Management Software & Tools
3. Documentation
Many IT professionals seem allergic to documentation. However, documentation helps to set expectations, track status, and communicate clearly within the patch management team and to other stakeholders.
Minimum Expectations
Adopt a basic patch management policy to establish baseline requirements for what to patch, patching process details, how to manage exceptions, and how to manage disruptions. A few weeks of negotiations can prevent more permanent grudges between departments.
Schedule patching and downtime to minimize impact and disruptions to operations. Determine the approval process, notices (internal, customer, etc.), approvals, and other documentation required when assets must be disabled or restarted to apply patches.
Generate reports to track what assets were upgraded/patched, exceptions, and any issues encountered. Good documentation makes the next patching cycle more efficient and helps prevent confusion during IT team transitions or incident response.
Assign ownership of patch management to an executive or manager and make it a key performance indicator. The process can be delegated or even outsourced, but ensure the designated executive or manager remains familiar with the results and the process.
Next Level Implementation
Patch management team definition and coordination is required at the enterprise level due to extra complexity. Patching may be performed by multiple patch management teams, cover diverse IT environments, and be forced to meet multiple compliance standards. Better patch management tools can help coordinate activities, classify assets, embed workflows, and enable comprehensive reporting.
Change management policies and reports will be needed to establish who notification requirements, how specific changes should be documented, what reports should be generated afterwards. Change management reports help to verify approved changes to critical assets as opposed to unauthorized and potentially malicious changes.
Pro Tips
Be detailed. Specify the members of the patch management team and their responsibilities clearly. Determine required notification processes in detail. If different processes or teams will be required for specific technologies (servers, cloud platforms, etc.) create exhibits with specific teams and requirements. However, be careful not to be so detailed that the IT team cannot flexibly make adjustments according to available resources and circumstances.
Publicize patching processes to make regular patching processes known and predictable for the organization. Users get used to the regular process and understand that normal patching isn’t typically disruptive or burdensome. This also adds credibility to urgent patches by making them seem even more important.
Take patching seriously and require the reporting of key metrics at the executive level such as the percent of devices patched, the time to patch for critical and non-critical patches, exception reports, etc.
4. Effective Patching Process
The key goals for patch management often conflict because quick application of patches to block security vulnerabilities can introduce disruption to operations and require already-committed IT resources. An effective patching process manages these conflicts to obtain the correct patches, ensure their effectiveness, and deploy them with minimal operational impact.
Minimum Expectations
Monitor for updates, especially the most critical assets. Many vendors offer email or similar mechanisms to notify teams of updates; for example, almost all organizations should directly or indirectly subscribe to Microsoft Technical Security Notifications.
Only use verified sources to download patches and updates. Some malicious hacking groups create websites and phishing attacks offering security updates embedded with malware.
Automate where possible. Small IT teams can be overwhelmed with the volume of updates on top of their day-to-day duties. Various vendors provide patch management tools that automate patching for specific assets (Windows machines, macOS, third-party software, networking equipment, etc.). Some vendors also offer free tiers for smaller companies. Before adopting a specific tool, verify what assets will be covered and test the tool to verify it will relieve and not burden the patch management team.
Some small businesses rely upon Microsoft/Apple automated patch management built into the operating system, but they need to recognize two key weaknesses. First, the organization will not have visibility into the status of the endpoint or when patches may fail or be delayed by users. Second, the OS may be a critical asset, but it is hardly the only asset that needs to be updated on a regular basis. An OS-centric update will neglect vital updates for Adobe, Google, Mozilla, Oracle, and many other vendors and applications. A more formal and centralized patch management tool or service is a superior option.
Prioritize patches based upon the risk of the asset and the criticality of the vulnerability being patched. Non-critical patches should be updated on a scheduled basis to minimize disruption.
Coordinate with operations or other stakeholders and follow careful communication and pre-approved procedures for patches that will cause operational disruptions.
Audit assets to locate failed patches or to recover devices or assets disrupted by an applied patch.
Report results as required by the patch management policy (see above) or basic KPIs. Update the asset list with the latest software/firmware versions to make future patching processes more efficient.
Next Level Implementation
Threat intelligence and government resources such as the NIST National Vulnerability Database should enhance vendor monitoring for vulnerability and patch announcements. These feeds provide additional context regarding actively exploited vulnerabilities and helps to prioritize patches that may be actively exploited or especially critical.
Leverage vendor relationships, register devices, and keep in contact with sales reps and suppliers to receive patch notices as quickly as possible. Some vendors have been known to provide direct notice to customers in advance of public vulnerability and patching announcements.
Centralize patch management to consolidate expertise, reporting, and eliminate redundancies (repetitive testing, waiting for redundant rollout, etc.). Centralized patch management can also create local network distribution points to reduce network and internet traffic from thousands of machines reaching out to a vendor’s site to download patches.
Patch priority, relevance and urgency can be more difficult to determine in enterprise environments. Diverse technology, version inconsistency between offices, and variations in the tech stack complicate the understanding of risk and how vulnerabilities might apply.
“It is hard for many enterprises to accept the fact they cannot patch everything,” explains Bob Kelly, director of product management at Flexera. “As little as one in 10 patches actually get deployed due to challenges in identifying, testing, and rolling out updates to the constant backlog of security updates coming on a daily basis.”
A patch management solution that integrates with risk values, threat intelligence, and CVSS 3.1 can help to automate or at least rapidly understand the risk profile of the vulnerability. Also, keep in mind that unapplied patches may be replaced by subsequent patches. An organization will need to make sure that obsolete patches are dropped from the queue.
Verify security control assumptions. Controls can mitigate vulnerabilities and decrease urgency, but those controls must be verified. For example, verify there is no overlooked access to a critical vulnerability on a database server that is supposed to be only accessible on an internal network. Bad assumptions can undermine the understanding of risk and leave critical assets more exposed than expected.
Test patches in a test environment to check for conflicts with other software, hardware, and IT architecture. If resources are constrained, focus on critical patches and critical systems. Testing is typically out of reach for smaller organizations, and can cause critical delays for high-risk patches.
Fortunately, many patch management tools and service providers include patch testing as part of their package – with some testing patches within hours of their release. While pre-tested patches may eliminate the risk of broad disruption, keep in mind that each organization’s architecture and assets will be unique and may experience unusual circumstances.
Limited patch rollout to specific devices. Staggered deployment allows the patch management team to observe results and lower the network bandwidth requirements of large scale patch deployments. Limited deployment can also limit the damage of unexpected conflicts from a patched vulnerability.
“Applications and computer systems are extremely complex, and there’s always a chance that a new update may create unintended problems,” cautions Lou Fiorello, vice president and general manager of security products at ServiceNow. “Roll out a new patch in a controlled environment before trusting it with an entire network.”
Next level-automated tools expand patch management to networking equipment, servers, IoT devices, and a wide variety of third-party software. However, different tools will have different management and integration capabilities so patch management teams may need multiple tools and still have some assets that require manual patching because they are not covered or because they require manual processes.
Redundant and failover systems provide a built-in test environment for patch deployment and for less disruptive patch roll-out. A web server’s failover clone can be patched and checked for potential conflicts. Once verified as successfully updated, the failover server can be deployed as the primary server and the former primary server can be updated and put into service as the failover system. Cloud, containers, and SaaS technologies often use this technique to maximize uptime.
Extend patching to AppBoM. Larger organizations with application development need to monitor open source code and libraries in their application bill-of-materials. The DevOps or DevSecOps teams may have an independent patch management process, but their own patch management reports should be reported to executives either separately or in a consolidated report.
Pro Tips
Patch with urgency. Mandiant estimates that more than 27% of disclosed vulnerabilities are exploited less than a month after the release of a patch. Lower risk systems can be patched with less urgency, but keep in mind that zero-day attacks or other undetected vulnerabilities can allow attacks on those lower risk systems to jeopardize systems of much higher value.
Patching priority does not equate to risk value. Some organizations exclusively patch Common Vulnerability Scoring System (CVSS) vulnerabilities with a score of 7 or higher, but attackers take advantage of this trend. “We are seeing an uptick in exploitation of vulnerability in the 5–7 range because hackers know that those scored 7 and higher are more likely to be mitigated faster,” warns Bob Kelly, director of product management at Flexera.
Organizations need to combine CVSS scores with risk ratings and known exploits. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) maintains a list of known exploited vulnerabilities. The list currently only shows about 900 vulnerabilities – a seemingly small number when more than 20,000 new vulnerabilities are identified each year – but it still requires good asset discovery and prioritization to target just that relatively small list.
Combining risk management, CVSS scores, and active exploitation will create a patching priority such as:
- Patch critical vulnerabilities with high CVSS scores with known exploits within 72 hours of release.
- Patch vulnerabilities with known exploits within 7 calendar days of release.
- Patch known vulnerabilities with high CVSS scores within 10 business days of release.
- Patch other updates within 30 days of release.
Hire expertise or outsource if patch management goals cannot be met using in-house resources. Patch management is one of the most widely outsourced functions for managed IT service providers (MSPs) who can patch a wide range of systems and software for competitive prices. Before signing a service agreement, companies should verify that the MSP will meet or exceed the patch management goals for patch deployment speed, minimized operational disruption, and reporting.
Also read:
- Is the Answer to Vulnerabilities Patch Management as a Service?
- Vulnerability Management as a Service (VMaaS): Ultimate Guide
- Best Third-Party Risk Management (TPRM) Tools
5. Mistake Management
Even the best plans, processes, and technology can fail. Organizations need to maintain disaster recovery plans and processes to recover systems in a broad range of scenarios. The elements of disaster recovery for patch management revolve around restoring systems and data.
Minimum Expectations
Backups should always be executed prior to patching to enable effective roll back or back-out of updates. Local system backups often will work, but in the event of a catastrophic patch failure, independent backups on different assets will be needed.
Next Level Implementation
Vulnerability scans should be conducted after patching to verify proper installation and to check for potentially undisclosed issues. Some patches have been known to cause conflicts with security tools or introduce new vulnerabilities.
Pro Tips
Periodic penetration tests should be conducted to verify that the controls protecting unpatched systems remain effective. Risk assessments assume controls continue to work; however, these assumptions should be tested.
Also read: 8 Best Disaster Recovery Solutions
Bottom Line: Patch or Pay
Organizations that execute patch management effectively will enjoy many benefits such as improved security, improved features, and avoiding costly incident responses. Effective patching policies prevent far more costly emergency IT responses that force overtime and additional costs for asset recovery.
More importantly, breaches can also trigger compliance violations that lead to lawsuits and regulatory fines. Many governments such as Australia and the United States list vulnerability patching as fundamental IT security practices so failure to patch will likely be viewed as negligence and punishable by stiff penalties.
Patch management can be boring because it is so fundamental. However, execution can be complex and many breaches reveal that many struggle to manage their processes well. Best practices can enable effective implementation of the patching process and should be adopted by organizations of all sizes.
Read next: Top Vulnerability Management Tools
This article was originally written by Drew Robb on October 21, 2022. It was updated by Chad Kime on February 24, 2023.
