A security information and event management system, or SIEM (pronounced "SIM"), is a security system that ingests event data from a wide variety of sources such as security software and appliances, network infrastructure devices, and many of the applications running in an organization.
A SIEM has two closely related purposes: to collect, store, analyze, investigate and report on log and other data for incident response, forensics and regulatory compliance purposes; and to analyze the event data it ingests in real time to facilitate the early detection of targeted attacks, advanced threats, and data breaches.
In the market for a SIEM solution? See our picks for Top 10 SIEM Products.
SIEM's security and threat management capabilities
When SIEM systems first started appearing, adoption was principally driven by large enterprises' compliance requirements. For that reason, most SIEM systems were tailored toward the requirements of these large organizations.https://o1.qnsr.com/log/p.gif?;n=203;c=204660766;s=9477;x=7936;f=201812281312070;u=j;z=TIMESTAMP;a=20392931;e=i
A much more significant driver for adoption today is the security and threat management capabilities that SIEM systems offer. There is a general acceptance that a well-resourced attacker – such as one backed by a nation state – can overcome any security system to break in to a corporate network. For that reason, many organizations are focusing their efforts on detecting attacks and security breaches as quickly as possible and then responding appropriately to minimize the harm they can do.
A SIEM system is a powerful tool that can be used to assist in early detection, so many small and medium-sized businesses that were not interested in adopting a SIEM for compliance purposes are now implementing them. This includes new deployments by organizations with limited security resources that need to improve monitoring and breach detection at the insistence of larger customers or business partners, according to Gartner.
SIEM vendors and service providers have been quick to respond with offerings that meet the needs of these smaller organizations and their more modest security resources.
Why do companies need SIEMs?
The reason that companies need SIEM systems to monitor logs and report suspicious events is that most organizations generate far too much event data for any human to be able to make sense of it.
To get an idea of the scale of the data involved, Gartner considers a small SIEM deployment to be one with up to 300 event sources, with events being generated at the rate of up to 1,500 events per second (EPS) and a data store of up to 800 GB.
Mid-sized deployments have up to 800 event sources, a sustained event rate of up to 7,000 EPS, and up to 8 TB of storage.
Very large deployments have thousands of event sources, and may generate more than 25,000 EPS, with a back store of over 50 TB.
A SIEM's ability to filter through all the data and only surface the most critical security issues helps make security more manageable. A SIEM also plays a key role in a number of compliance regulations.
SIEM market share
At about 2.5%, SIEM accounts for a small but increasing proportion of global security spending. In total, organizations spent about $2.4 billion on SIEM out of a total security spend of about $98 billion in 2017, but this is expected to rise to $2.6 billion in 2018 and $3.4 billion in 2021, according to Gartner.
How a SIEM system works
A SIEM system is essentially a specialized Big Data analysis system that seeks to generate useful insights from the mass of events and other data that it ingests and stores. The key source of data is the logs generated by systems, including your servers and security appliances, but SIEMs can ingest a variety of other sorts of data, including NetFlow and network packets, as well as contextual information about users, assets, threats and vulnerabilities that can be found inside or outside your organization.
This data from diverse sources must then be "normalized," or reformatted, so that the SIEM can make sense of it.
Once the ingestion and normalization has been carried out, the SIEM can get down to its bread-and-butter work:
- Ingestion and interpretation of logs
- Threat intelligence feeds
- Correlation and analytics
- Security alerts
- Data presentation
A key differentiator between SIEM tools is the number and variety of log sources that they can connect to out of the box for data aggregation purposes. Although it is usually possible to build a connector to an individual device or application, this can be costly and time-consuming and therefore impractical for more than a handful of log sources.
"It is of key importance that as many of your log sources as possible are supported. That's because making your own connectors doesn't scale well," said Oliver Rochford, a former research director at Gartner.
If you have your own custom apps, you will have to make your own connectors for them. But ensuring that your commercial apps are supported could be the difference between a SIEM project succeeding and failing.
The likely sources of logs that you will want a SIEM system to ingest include:
- Intrusion detection systems/intrusion prevention systems (IDS/IPS)
- Data Loss Prevention (DLP) systems
- Anti-virus and other endpoint security software
- Unified Threat Management (UTM) systems
- VPN concentrators
- Web filters
- Honeypot or deception systems if you have them
- Routers and switches
- Domain controllers
- Wireless access points
- Application servers, intranet application and databases
Many companies only make use of the threat intelligence feed or feeds included with the SIEM product or service they buy. However, commercial feeds from third parties and open source threat intelligence feeds are also available. Commercial and open source threat intelligence feeds are valuable because research shows that their content does not overlap to a high degree. The more information a SIEM has about security threats, the more likely it is to detect them.
Nonetheless, for simplicity or for financial reasons, some may elect to only utilize those feeds included with their SIEM. For that reason, it is important to establish which feeds are included and which other commercial and open source feeds are compatible with the SIEM product. OpenIOC is a framework or standard that ensures SIEM compatibility with many feed sources.
Correlation and analytics is the core of SIEM technology, and it involves tying together different occurrences reported in logs to spot the indications of a compromise. One example: a port scan followed by user access to certain types of data. It is important that analytics functionality is presented in a way that you can use effectively; you should check this thoroughly before any purchasing decision.
Key to this ability to spot threats is the use of correlation rules, a core set of which may be provided with the SIEM system but which administrators can add to. For example, one correlation rule might be that if four or more failed login attempts occur from the same IP address using different usernames within a period of 15 minutes, and this is followed by a successful login from that IP address to any device on the network, then an alert should be issued. In this case, the SIEM system's correlation engine is being put to work to detect a pattern of behavior (failed login attempts followed by a successful one) that may indicate a brute force attack has been successful.
Those security alerts are perhaps the most important feature of a SIEM tool, the ability to use correlation rules to alert security staff as quickly as possible about possible security incidents. Alerts can be displayed on a centralized dashboard or provided in a number of other ways, including via automated emails or text messages.
One potential problem with a newly installed SIEM system is that it may issue too many false positive alerts, taking up security staff time unnecessarily. It's important to look at all pre-configured rules supplied with the system and disable any that are not applicable to your specific network. Some disabled rules may need to be replaced by new rules that have to be rewritten from scratch to suit your particular network architecture.
An important function of a SIEM system is to make the interpretation of data from multiple sources easier by presenting it in the form of easily comprehensible graphics on a security dashboard display. This is far easier to digest at a glance, compared to having to wade through dozens of logs looking for anomalies. It's one more way a SIEM system can improve incident response and efficiency.
SIEM technology is commonly used to collate events and logs and to generate compliance reports to meet specific compliance requirements, eliminating tedious, costly and time-consuming manual processes. Some offer integration with the Unified Compliance Framework, enabling a "collect once, comply with many" approach to compliance reports. Having a SIEM is a core part of a number of compliance regimes, such as PCI-DSS and ISO 27001, the core ISO spec for security systems management.
Advanced SIEM capabilities: UEBA and SOAR
Many SIEM systems go beyond the basic to incorporate advanced features and capabilities. Two in particular are User and Entity Behavior Analytics (UEBA) and Security Orchestration Automation and Response (SOAR).
User Behavior Analytics (UBA) and User and Entity Behavior Analytics (UEBA)
Three quarters of all network breaches are carried out using compromised user authentication credentials (such as passwords), so many SIEM systems have developed ways to spot unusual or anomalous user or entity behavior that could be indicative of a breach.
The idea behind this is simple: If a given user "Brian" who works in marketing usually logs in from the marketing department every weekday morning between 9 a.m. and 9.30 a.m. and logs off at around 5 p.m., then it is very suspicious if Brian suddenly tries to log on to the network at 3 a.m., apparently from the IP address of a workstation in the finance department. Such suspicious behaviour would result in an alert.
Security Orchestration Automation and Response (SOAR)
SOAR heralds the start of next-generation SIEM systems, with the ability not just to issue alerts when suspicious activity or threats are detected, but also to respond by carrying out actions to mitigate the threat automatically.
For example, a routine vulnerability scan combined with a threat intelligence feed could show that a particular server on a network is vulnerable to a new exploit. A SOAR system could orchestrate and automate a third-party update system to patch the server, or perhaps use other tools to mitigate the threat automatically.
Artificial intelligence and big data analytics
Vendors are beginning to introduce artificial intelligence and deep learning capabilities into their SIEM systems in a number of ways. These include automatic rule generation, anomalous behavior detection and advanced statistical analysis, all designed to spot threats more quickly and with fewer false positives.
However these innovations are in the early stages, and it has not yet been clearly demonstrated that they provide any real benefits over existing SIEM methods.
How to select a SIEM product
With a wide variety of SIEM systems for you to choose from, the starting point for product selection is establishing what you hope a SIEM will provide you with, and what your particular needs are.
For example, if your primary driver for buying a SIEM is compliance, then you are likely to value systems that offer detailed reporting capabilities. If you want to set up a security operations center (SOC), then a more security-focused product will be more suitable. If you want help spotting new threats, then a product with better data visualization tools and search capabilities will be more useful.
Your organization's size is another important determining factor: If you are generating 100,000 events per second (EPS), then you will be restricted to some of the very largest capacity SIEM systems.
A recent survey of security information and event management (SIEM) users in 559 large organizations across the U.S. carried out last year found that 84 percent said their SIEM is important, very important, or essential to their incident response process.
But the survey, conducted by the Ponemon Institute and sponsored by Cyphort, a security startup acquired by Juniper Networks in 2017, also found that just 48 percent are satisfied with the intelligence they receive. This may be due to the fact that they are too focused on one feed, get too many alerts, or that companies are expecting too much from a SIEM.
Staffing and integration challenges may also lie behind SIEM dissatisfaction where it exists.
SIEM staffing needs
The Ponemon Institute survey discovered that 25 percent of companies' investment in SIEM is related to the initial purchase of the software. The remaining 75 percent goes to installation, maintenance and staffing. It should be noted that 78 percent had one or less full-time staff member assigned to SIEM administration. That's a major reason why 64 percent spent over $1 million a year for external consultants and contractors to help with SIEM configuration and management.
68 percent of respondents said they would need additional staff to maximize the value of SIEM.
Clearly, if you are considering purchasing a SIEM system, you should factor in the staffing and maintenance side in the financial equation. It's not enough just to count the purchase cost of the software and hardware because you need to consider the cost of skilled personnel and the increase to your security operations budget to ensure the necessary feed subscriptions can be purchased and integrated.
An alternative approach, of particular interest to smaller companies that cannot justify providing a SIEM system with 24-hour staff coverage, is to employ a managed security service provider (MSSP) that can provide "SIEM as a service" by monitoring your systems for you and, in some cases, act on alerts by determining and carrying out suitable actions on your behalf.
How to implement a SIEM
Implementing a SIEM can be a lengthy and expensive process, and you may find it useful to use consultants or vendor-supplied professional services to help ensure implantation is carried out efficiently and successfully. The end result will be a system that better meets your needs.
Basic steps include:
- Determining the system architecture, including dashboard and reporting systems, indexing and storage systems, and log collections systems
- Choosing appropriate hardware based on factors that include the projected volume of log data to be collected (measured in events per second) and the number of log sources
- Establish your storage requirements and how that will be provided, along with suitable storage network infrastructure to access it
- Installation of servers and software or appliances
- System configuration, including setting up log ingestion using supplied or custom connectors, setting up dashboards and scheduling reports, configuring correlation rules, and enabling all required alerts
SIEM wish list
Organizations intending to implement a SIEM system would be wise to learn from those who are already running it. Those using SIEM tools currently have some definite ideas about what they would like their systems to be able to achieve.
According to Ponemon, 71 percent of those surveyed would like to be able to automate specific SIEM-generated tasks in order to allow response teams to focus on priorities. 70 percent want their SIEM to generate fewer alerts that are more accurate, prioritized and meaningful. And 54 percent said they get too much low-level data and too many alerts, making it hard to focus on what matters.
"The research data from the Ponemon Institute is consistent with the feedback we've been hearing from many organizations across the U.S. in terms of the problem with SIEMs," Cyphort's Chief Marketing Officer said when the research was published. "The quantity of data is too high, while the quality of the data is too low. And there is inadequate staff to minimize that noise and maximize the underlying value."
These comments are important, as they indicate that while SIEM systems do have the potential to make your company more secure, a successful implementation that maximizes the system's value is a considerable challenge.
This updates a June 5, 2017 article