Security information and event management (SIEM) is the technology that can tie all your systems together and give you a comprehensive view of IT security.
IT security is typically a patchwork of technologies – firewalls, intrusion prevention, endpoint protection, threat intelligence and the like – that work together to protect an organization's network and data from hackers and other threats. Tying all those disparate systems together is another challenge, however, and that's where SIEM can help.
SIEM systems manage and make sense of security logs from all kinds of devices and carry out a range of functions, including spotting threats, preventing breaches before they occur, detecting breaches, and providing forensic information to determine how a security incident occurred as well as its possible impact.
What is SIEM?
A security information and event management, or SIEM solution (pronounced "SIM"), ingests log data from a wide range of network hardware and software systems and analyzes that data in real time. Its purpose is to correlate events and spot individual anomalies or patterns of behavior that may indicate a security breach – using intelligence feeds to ensure that it is aware of new threats as they emerge – and to present log data in a manageable and easily understood form so that it can be interpreted effectively by security staff.
SIEM tools are also used to collect log information from security and other systems to generate reports for compliance purposes. Security information and event management is sometimes also known as security event and information management.
Two related activities are SEM (security event management) and SIM (security information management). These are both subsets of SIEM. In general, SEM is concerned with real-time monitoring of logs and correlation of events, while SIM involves data retention and the later analysis and reporting on log data and security records. This is often carried out as part of a forensic analysis to establish how a security breach occurred, what systems and data may have been compromised, and what changes need to be made to prevent a similar breach. Most modern SIEMs can be used to carry out both SEM and SIM.
SIEM and the midmarket
In the past, SIEM products were generally limited to large enterprises, but they are now becoming accessible to medium-sized organizations as well, said Oliver Rochford, a former research director at Gartner and now a security evangelist at cybersecurity vendor RiskSense.
One reason that SIEM products were mostly used only by large enterprises is that you need one or two people to oversee them 24-7, and only large companies typically have these kinds of human resources. But medium-sized companies can use either a managed service or oversee a SIEM system during office hours and rely on a managed service to provide "out of hours" coverage, Rochford suggested.
Another reason that the appeal of SIEMs has broadened is that previously the main driver for adoption was compliance, an issue more likely to affect larger companies. While compliance is still an important factor, Rochford said threat management is now a bigger driver.
"Look at ransomware; that's a threat that midsized companies are very interested in detecting," he said. "Ransomware is typically very compact, and then it connects to a C&C (command and control) center. So you may be able to detect a phishing email that delivers it, or its communication, or indicators of a compromise like new processes starting. A SIEM will allow you to centralize and review this information and maybe detect the ransomware."
SIEM tools: key features to look for
SIEM is a very broad term, and SIEM tools from different vendors have different feature sets, strengths and weaknesses. But in general most SIEM tools will have variations of the following features:
- Ingestion and interpretation of logs
- Connection to updated threat intelligence feeds
- Correlation and analytics
- Advanced profiling
- Security alerts
- Data presentation
A key differentiator of SIEM tools is the number and variety of log sources that they can connect to out of the box for data aggregation purposes. Although it is usually possible to build a connector to an individual device or application, this can be costly and time consuming and therefore impractical for more than a handful of log sources. Certain vendors such as Splunk are notable for the large number of applications that they can ingest data from.
"It is of key importance that as many of your log sources as possible are supported. That's because making your own connectors doesn't scale well," Rochford said.
If you have your own custom apps, you will have to make your own connectors for them. But ensuring that your commercial apps are supported could be the difference between a SIEM project succeeding and failing. If you use a wide variety of applications, you may need to look at a vendor such as Splunk.
Many companies only make use of the feed(s) included with the SIEM product or service they buy. However, commercial feeds from third parties and open source threat intelligence feeds are also available. Commercial and open source threat intelligence feeds are valuable because research shows that their contents do not overlap to a high degree. The more information a SIEM has about security threats, the more likely it is to detect them.
Nonetheless, for simplicity or for financial reasons, some may elect to only utilize those feeds included with their SIEM. For that reason, it is important to establish which feed(s) are included and which other commercial and open source feeds are compatible with the SIEM product. OpenIOC is a framework or standard that ensures SIEM compatibility with many feed sources.
This is the bread and butter of SIEM technology, and it involves tying together different occurrences reported in logs to spot the indications of a compromise. One example: a port scan followed by user access to certain types of data or user entity behavior that can indicate an internal threat. It is important that analytics functionality is presented in a way that you can use it effectively; this should be checked thoroughly before any purchasing decision.
All SIEMs carry out correlation and analysis, but advanced profiling is less common (although it is becoming increasingly prevalent). This can be implemented in many ways. But in essence, it works by establishing baseline or "normal" behavior for a number of characteristics on a network. SIEMs traditionally do some form of correlation analysis. But profiling allows you to spot deviations from the norm. It achieves this by using behavioral analytics. "In many cases, such a deviation indicates something bad, or at least suspicious activity," said Rochford.
Perhaps the most important feature of a SIEM tool is the ability to use the features described above to alert security staff as quickly as possible about possible security incidents. Alerts can be displayed on a centralized dashboard (see below) or provided in a number of other ways including via automated emails or text messages.
An important function of a SIEM system is to make the interpretation of data from multiple sources easier by presenting it in the form of easily comprehensible graphics on a security dashboard display. This is far easy to digest at a glance, compared to having to wade through dozens of logs looking for anomalies.
SIEM technology is commonly used to collate events and logs and to generate compliance reports to meet specific compliance requirements, eliminating tedious, costly and time-consuming manual processes. Some offer integration with the Unified Compliance Framework, enabling a "collect once, comply with many" approach to compliance reports.
Tool trends and innovation
Not surprising in an area as critical as IT security, SIEM vendors continue to innovate. Here are some features that many vendors have already implemented or plan to:
The level of automation in SIEM tools today varies considerably. It may include basic automation such as generating automatic notifications when certain events are triggered or thresholds are passed, or more advanced functionality such as the automatic gathering of intelligence after an incident is detected. The most advanced form of automation involves triggering automated responses to threats that are detected.
Today this form of automation automated responses to detected threats is comparatively rare because of worries about the disruption that could be caused in a production environment if a false positive is triggered. For that reason, it tends only to be used by organizations that want to adopt the very highest security posture, but in the future it is likely that automated responses may become the norm when faced with sophisticated attacks from hackers using automated attack tools.
"There are still inhibitions about using automated responses because of worries about what it could do to a production environment, so it is primarily used by early adopters or by companies that want to adopt a very high security posture," Rochford said. "But I do think that automated responses are going to be necessary so its use will become more common."
Artificial intelligence and machine learning:
AI and machine learning go hand in hand with automated response and the ability to react immediately to detected threats, and it is likely that they will become increasingly important features of SIEMs in the future. However, in the near term most SIEM systems will only use artificial intelligence and machine learning as a complement to human oversight rather than as a fully automated alternative to a human-managed system.
Rochford warns that AI is not a panacea that will solve all security problems. "The message is don't have too high expectations about artificial intelligence. It can certainly do some things that humans can't, but it can't do everything."
A recent survey of security information and event management (SIEM) users in 559 large organizations across the U.S. found that 84 percent said their SIEM is important, very important or essential to their incident response process. But the survey, conducted by the Ponemon Institute and sponsored by Cyphort, also found that just 48 percent are satisfied with the intelligence they receive.
This may be due to the fact that they are too focused on one feed, or that companies are expecting too much from SIEM. While it can provide valuable insight, as Rochford said, it is no panacea. But staffing and integration challenges may also lie behind SIEM dissatisfaction.
The Ponemon Institute survey discovered that 25 percent of companies' investment in SIEM is related to the initial purchase of the software. The remaining 75 percent goes to installation, maintenance and staffing. It should be noted that 78 percent had one or less full-time staff member assigned to SIEM administration. That's a major reason why 64 percent spent over $1 million a year for external consultants and contractors to help with SIEM configuration and management.
68 percent of respondents said they would need additional staff to maximize the value of SIEM.
"This data indicates that the demand for trained security analysts exceeds the supply of skilled talent available to fill these positions," Ponemon Institute chairman and founder Dr. Larry Ponemon said in a statement.
Clearly, those looking to purchase SIEM should factor in the staffing and maintenance side to the financial equation. It’s not enough to purchase the software. Those planning to do so have to be committed to harnessing it to greatest effect. That entails adding skilled personnel and increasing the security maintenance budget to ensure the necessary feeds are integrated.
The SIEM wish list
Organizations intending to implement SIEM would be wise to learn from those who are already running it. Those using SIEM currently have some definite ideas about what they would like their systems to be able to achieve.
According to Ponemon, 71 percent of those surveyed would like to be able to automate specific SIEM-generated tasks in order to allow response teams to focus on priorities. 70 percent want their SIEM to generate fewer alerts that are more accurate, prioritized and meaningful. 54 percent said they get too much low-level data and too many alerts, making it hard to focus on what matters.
"The research data from the Ponemon Institute is consistent with the feedback we've been hearing from many organizations across the U.S. in terms of the problem with SIEMs," Cyphort Chief Marketing Officer Franklyn Jones said in a statement. "The quantity of data is too high, while the quality of the data is too low. And there is inadequate staff to minimize that noise and maximize the underlying value."
Bear these points in mind during product selection, implementation and configuration. Due to the consequences of an undetected breach, money spent early in the project to alleviate such ills may be well spent.
How SIEM is implemented
According to John Marshall, vice president of technical services at STEALTHbits Technologies, SIEM adoption was initially driven by the need for a long-term archive for log files, not as a security monitoring solution per se. This has given rise to the commonplace "ingestion-volume" approach to licensing, which most vendors follow. However, this approach can suffer under high data volumes.
Trying to unite SIEM with multiple additional security platforms and threat intelligence feeds can lead to several negative consequences. Notably, operational cost and complexity will rise. But potential buyers should also understand that it isn't the volume of data fed into the system that counts – it is the quality. Under a high volume of poor data, costs will mushroom but effectiveness will remain low.
"Vendor differentiation needs to be driven by a focus on new use case-centric capabilities around improving the quality of data inputs and for addressing the challenges of ongoing data management," Marshall suggested.
Part of the implementation equation is who is going to do it and how it is going to be run. Some decide to implement SIEM in-house and manage it entirely using internal resources. This option works for those already possessing sufficient quantity of well-trained and experienced security resources. Those lacking such resources but who wish to host SIEM internally, may be wise to outsource the management functions, and in some cases event monitoring too, to alleviate the load. But for those needing SIEM but unable to provide the necessary personnel resources, outsourced SIEM in the cloud might be the smarter course. Here, the provider does the management and event monitoring and hosts the system in the cloud.
SIEM implementation tips
David Humphrey, senior enterprise security architect at Harvard Pilgrim Health Care, recommends that anyone embarking upon SIEM should first set the correct level of expectations on just what the project is. He likened it to being comparable to replacing the piston rings in the engine of a car.
Humphrey said users need to be aware of factors such as:
- All Unix systems need a centralized logging system to be developed for the SIEM
- Oracle databases require a great many configuration steps
- Each web server needs a new process installed to monitor the web logs
- Taking logs from cloud resources on AWS involves another complex series of configuration procedures
- SIEM requires highly skilled IT personnel resources
- Patching, hardware refreshes and overall change management have to be coordinated with SIEM
"It will take time and effort to get things set up to log to the SIEM, and this is going to be a manpower initiative proportional to the complexity of your organization," said Humphrey.
Those evaluating SIEM vendors are urged to verify that they adhere to the requisite security standards and certifications. As needs will vary depending on the industry and region, certain vendors may be favored over others based on their security credentials. Applicable security qualifications to consider include Federal Information Processing Standard (FIPS) 140-2, Common Criteria for Information Technology Security Evaluation (CC), Payment Card Industry Data Security Standards (PCI DSS), NIST SP 80092, NIST SP 80053, NIST SP 80082, NERC CIP007, NERC, GBLA, FISMA, Sarbanes-Oxley (SOx), GPG13, the Health Insurance Portability and Accountability Act (HIPAA), and ISO 27001.
Pricing models vary considerably from vendor to vendor. Some are based on the amount of data ingested (GB/s) and/or security events correlated per second (EPS). Others offer perpetual or annual term licenses, monthly or annual subscriptions, or charge based upon model utilized appliances and servers. And some provide free open source versions of SIEM but charge for support.
Top SIEM vendors
For our comprehensive analysis of SIEM solutions, see Top 10 SIEM Products.
Drew Robb also contributed to this report.