How to Run a Threat Hunting Program

Share it on Twitter  
Share it on Facebook  
Share it on Linked in  

Every company has cybersecurity systems in place to try to keep attackers off their networks and to protect their data. But skilled attackers can and do evade these security systems with advanced persistent threats (APTs), and when that happens it takes an average of 170 days before these APTs are detected, according to the Ponemon Institute.

These attacks are the driver for a different, complementary approach to security: threat hunting. The idea behind threat hunting is that the corporate IT infrastructure may already have been compromised, and therefore resources should be dedicated to searching it to uncover existing threats on a continuous basis.

So threat hunting can be defined as a proactive and iterative scouring of IT resources, including networks, datasets, servers, applications and endpoints carried out by skilled security staff to detect any suspicious, unusual, or obviously malicious activities that have not been detected by the organization's automated security systems.

To use a policing analogy, a threat hunter is like a beat cop who gets to know his neighborhood and is constantly looking around to spot anything out of the ordinary that may be the sign of undesirables sniffing around, of an impending break in, or of a crime that is in the process of being carried out.

Threat hunting methodologies

Establishing threat hunting goals

It may sound obvious, but the first stage in any threat hunting program is establishing "prioritized intelligence requirements," or PIRs, which is another way of saying that there is a need for figuring out your hunt goals, or what sort of threats you should be looking for.

Every organization is different, so it makes sense to work out what kinds of threats your organization should be most concerned about. This is generally something that should be discussed with stakeholders and C-level executives, and since the threats may change over time it should be revisited frequently.

One approach is to consider threats to the company's "crown jewels," which may be research data, customer lists, or production information. Then consider how these things could be compromised. Much of IT security has become automated, but this is one area where good old-fashioned critical and creative thinking has a major role to play.

Another approach is to use external threat intelligence sources so see what attackers are doing elsewhere to consider if any of these advanced threat activities could affect your organization.

Yet another is to use knowledge of previous security incidents that have affected your organization to consider if these incidents could happen again.

Formulating a threat hunting hypothesis

The next stage is to formulate a hypothesis about a threat, say attackers are exfiltrating your proprietary research data, or that they have successfully launched spearphishing attacks against one or more employees and have compromised their endpoints. The purpose of the threat hunt is then to prove or disprove the hypothesis.

Identifying indicators of compromise

To prove or disprove the hypothesis, it is necessary to define Indicators of Compromise (IoCs). These will depend on the chosen hypothesis. In the case of data exfiltration, they could include things like unusual network traffic flows, unexpected database reads, or anomalous user account activity. For spearphishing attacks IoCs might include abnormal document names, word processors spawning command line tools such as PowerShell, or PowerShell downloading executables or spawning processes itself.

Threat hunting techniques

In order to spot IoCs and identify the threat, skilled threat hunters employ a range of techniques when they analyze data sources such as firewall logs, SIEM and IDS alerts, DNS logs, file and network data, authentication systems, and other sources.

  • Baselining: One of the most important threat hunting techniques is baselining. This involves establishing what is (or should be) normal, and then examining deviations from this. For example, in the spearphishing example above, it might be sensible to examine which staff members are likely to use PowerShell, how frequently it is used and what their activity looks like when they use it. Armed with this data, it may be possible to identify anomalous PowerShell use and investigate further.
  • Stack counting: Stack counting or stacking is conceptually similar to baselining in that it looks for anomalies. In the case of stack counting, however, it involves examining the values of a particular type of data and putting similar values into "stacks." This may reveal that 99% of the data goes into a small number of tall stacks, with the outliers deserving closer scrutiny. There may be many reasons for the outliers, so a threat hunter needs to understand the data and use creativity and imagination to correctly identify the cause of the outlying data – which may or may not indicate a threat.
  • Clustering: This is a statistical technique, sometimes enhanced by machine learning systems, that identifies clusters of similar data points based on certain characteristics. This can enable a threat hunter to spot similarities and unexpected correlations in certain types of network (or other) activities that may warrant further investigation. Machine learning systems generally depend on being trained with known data, and then use this training to classify unknown data and spot anomalies.
  • Grouping: This goes a stage further than clustering by examining suspicious data sets to try to identify the underlying cause. For example, an analyst may group occurrences of certain types of data into specific time windows, which may isolate reconnaissance commands such as port scans that were initiated on a certain date.

Automated vs. manual threat hunting

Threat hunting relies on human analysts to piece together information and events to correctly identify threats. It needs knowledge and creativity and an understanding of how an attacker is likely to operate, in the same way that effective penetration testing requires a skilled human agent.

But threat hunting is also a process-driven big data security analytics exercise, and there is no doubt that a human threat hunter can be assisted hugely by automated threat hunting software, sometimes employing artificial intelligence,  which can do a huge percentage of the drudge work very quickly. For example automated event analysis can go through millions of logs in a very short space of time to identify specific anomalous events and triggering alarms.

But what all automated threat hunting solutions have in common is that – like automated penetration testing tools – they need to be guided and overseen by humans who can interpret their results and decide how they can best be put to use.

Automated threat hunting products

Although a relatively new area, there are a number of automated threat hunting platforms to choose from, including:

  • Carbon Black (formerly Bit9)
  • CrowdStrike
  • Cybereason
  • Darktrace
  • Endgame
  • ExtraHop Networks
  • Sqrrl (now owned by Amazon)
  • Vectra

Threat hunting tools

Distinct from software designed to automate much of the threat hunting process, there are also a huge range of software tools threat hunters use to wade through and make sense of the vast swaths of data they encounter and help them in their hunt for threats. These are different from breach and attack simulation (BAS) and endpoint detection and response (EDR) solutions, which are designed for reactive security staff.

  • Spreadsheets: The simplest threat hunting tool is the humble spreadsheet, which many threat hunters use to help them when carrying out a stack counting exercise to manage the numbers and sort them so that outliers can easily be spotted.
  • Security monitoring tools: Conventional security products such as firewalls, antivirus software, data loss prevention systems, and network intrusion detection systems are all used by threat hunters to help reveal indicators of compromise.
  • Statistical analysis tools: These use mathematical patterns to spot anomalous behavior in data, which the threat hunter may then decide warrants further investigation.
  • Intelligence analytics tools: These tools help threat hunters visualize data with interactive charts and graphs that make it easier to spot previously hidden correlations and connections between entities, events, or data.
  • SIEM systems: Security Information and Event Management (SIEM) solutions are used by threat hunters as well as reactive security staff to make sense of the vast amounts of log data that many organizations generate and to surface suspicious activity.
  • User and entity behavior analytics tools: UEBA tools can help threat hunters spot anomalous behavior.
  • Threat intelligence resources: As well as tipping threat hunters off about new threats to look for and techniques that attackers are adopting, threat intelligence resources also give details of specific executables or malware hashes to look for and malicious IP addresses to be wary of.

Threat hunting teams: Requirements and training

Most companies only carry out threat hunts when they know they have been compromised in some way, according to the SANS Institute, but introducing some form of routine threat hunting capability is becoming increasingly common. Many organizations lack the resources or in-house skills to carry out their own threat hunts, and choose to outsource threat hunting to specialized security companies.

The reason for this is that threat hunting requires a wide range of skills, and these include:

  • Cyber security experience: knowledge of network and endpoint security, data analysis, intelligence analysis, forensic science, malware reverse engineering
  • Understanding of attack methods: this includes past and current attack methodologies and trends in general, and the tactics, techniques and procedures (TTPs) associated with specific attackers or hacker groups
  • Operating system and network protocol knowledge: these should include good Windows and Linux skills as well as a deep knowledge of TCP/IP
  • Programming skills: at least one scripting language and compiled languages such as C or C++, and hunters also need to know how to automate tasks, parse logs, and carry out data analysis tasks

Many security experts say the best way to train as a threat hunter is simply to start hunting, gaining skills and confidence as you go.

However, there are an increasing number of threat hunting training courses available from organizations such as SANS, as well as qualifications such as Certified Cyber Threat Hunter (CCTH) from independent training organizations.