data fusion process refinement in intrusion detection alert correlation systems

DATA FUSION PROCESS REFINEMENT IN INTRUSION DETECTION ALERT CORRELATION SYSTEMS A Thesis Presented to The Graduate Faculty of The University of Akron In Partial Fulfillment of the Requir

DATA FUSION PROCESS REFINEMENT IN INTRUSION DETECTION ALERT

CORRELATION SYSTEMS

A Thesis Presented to The Graduate Faculty of The University of Akron

In Partial Fulfillment

of the Requirements for the Degree

Master of Science

David Sheets

DATA FUSION PROCESS REFINEMENT IN INTRUSION DETECTION ALERT

_ _

ABSTRACT

Computer systems are getting larger in size, contain a greater variety and volume of data,

and communicate personal and confidential information, making security critical as well as

making them appealing targets for malicious activities The need to keep these systems secure

has been approached from several different aspects, one of which is the employment of intrusion

detection systems An evolution of the intrusion detection system occurs in alert correlation

systems, which take raw alerts from numerous sensors within a network and generate broader

situational awareness by combining the individual findings of each sensor into a bigger picture

state of the system This study looks at improving the ability of an existing alert correlation

system to pull all the relevant pieces of an intrusion into that picture in order to further reduce the

output, enabling quicker analysis by a system administrator Through experimentation and

analysis, the benefits of utilizing the look-ahead system have demonstrated an ability to decrease

the total number of alerts in the system, thereby reducing the work-load of system administrators

by increasing the ability of the system to reduce the overall number of alerts the administrator

must analyze

ACKNOWLEDGEMENTS

It is with deep gratitude that I thank the people who have helped make accomplishing this

thesis possible

I would like to thank my wife Kristen, who for so long put up with me paying more

attention to the thesis and academic works than to her Her patience and contributions were

instrumental to reaching completion

Dr Xuan-Hien Thi Dang, my thesis advisor, who worked with me through the arduous task of completing a master’s degree while working full-time (and often greater than full-time)

Her advice provided the key elements that allowed the research and work combine smoothly into

this thesis

Dr Yingcai Xiao and Dr Zhong-Hui Duan, who took the time out of their busy

schedules near the end of the semester to provide invaluable feedback This allowed the paper to

take shape and to become a complete work Their efforts and dedication have provided the

critical finishing touches

I would also like to extend my appreciation to BGI-LLC and my professional colleagues

who allowed me the time and resources to complete this endeavor If not for their commitment, I

would have never been able to dedicate myself to this achievement

TABLE OF CONTENTS

Page

LIST OF TABLES viii

LIST OF FIGURES x

CHAPTER I INTRODUCTION 1

1.1 Statement of the Problem 1

1.2 Importance of the Study 2

1.3 Objectives 3

1.4 Contributions 5

1.5 Organization of the Thesis 6

II BACKGROUND OF THE STUDY 7

2.1 Intrusion Detection 7

2.2 Data Fusion 8

2.3 Alert Correlation 10

2.4 Data Sets 11

2.5 Alert Message Standardization 11

2.6 Correlation Engine Framework (STAT) 12

2.6.1 Data Fusion and Alert Correlation Concepts 12

2.6.2 Correlation Framework 14

2.6.3 AlertSTAT Framework 17

2.6.4 Limitations 20

III ALERT CORRELATION WITH LOOK-AHEAD 25

3.1 Correlation Process 25

3.2 Dynamic Time Window 26

3.3 Process refinement through a look-ahead 27

3.3.1 Algorithm 28

3.3.2 Applying the Dynamic Time Window 30

3.3.3 Providing Early Notification 31

3.3.4 Look-Ahead System 31

3.3.5 Benefits for process refinement 32

3.4 Examples 33

3.5 Example Output 38

IV RESULTS AND ANALYSIS 45

4.1 Experiments 45

4.1.1 Data Sets 45

4.1.2 Execution 47

4.2 Results 47

4.3 Analysis 60

V SUMMARY 63

5.1 Conclusion 63

5.2 Future Work 63

5.2.1 Improvements to the look-ahead 63

5.2.2 Further process refinement 64

5.2.3 Visualization 64

5.2.4 STAT Compiler Updates 65

BIBLIOGRAPHY 66

LIST OF TABLES

2.6-1 Output from static window example in Figure 2.6-4 23

3.4-1 Output from dynamic window example in Figure 3.4-1 34

3.4-2 Time window sizes in Figure 3.4-1 35

3.4-3 Output from extended static window example in Figure 3.4-2 38

4.1-1 Test platform 45

4.1-2 Statistics of available data sets 46

4.2-1 Overall results for full data set with default 120 second time windows 48

4.2-2 Overall results for full data set with session time window of 1,200 seconds 48

4.2-3 Session results for full data set with 120 second time window 49

4.2-4 Session results for full data set with 1,200 second time window 49

4.2-5 One2Many results for full data set with 120 second session time window 50

4.2-6 One2Many results for full data set with 1,200 second session time window 50

4.2-7 Many2One results for full data set with 120 second session time window 50

4.2-8 Many2One results for full data set with 1,200 second session time window 51

4.2-10 False positive predicitions on 1,200 second session time window 53

4.2-11 Results for subset 1 54

4.2-12 Results for subset 2 55

4.2-13 Results for subset 3 56

4.2-14 Results for subset 4 57

4.2-15 Results for subset 5 58

4.2-16 Results for subset 6 59

4.2-17 Results for subset 7 60

LIST OF FIGURES

2.2-1 Data fusion example using alerts 9

2.6-1 Alert correlation concept associated with data fusion 13

2.6-2 The AlertSTAT system implementation 19

2.6-3 Basis for correlation examples 22

2.6-4 Correlation example with static time window 23

3.3-1 Pseudo-code for calculating the dynamic window extension 29

3.3-2 Pseudo code for the window timer calculation 30

3.3-3 Look-ahead depth and technique used 32

3.4-1 Correlation example dynamic window with look-ahead depth of 2 34

3.4-2 Correlation example with extended static windows 37

3.5-1 Example output 40

3.5-2 Example IDMEF correlation alert 41

3.6-1 Example affect of a look-ahead false positive 43

4.2-1 Stacked plot of reduction with 120 second session time window on full data 51

4.2-3 Stacked plot of reduction in subset 1 54

4.2-4 Stacked plot of reduction in subset 2 55

4.2-5 Stacked plot of reduction in subset 3 56

4.2-6 Stacked plot of reduction in subset 4 57

4.2-7 Stacked plot of reduction in subset 5 58

4.2-8 Stacked plot of reduction in subset 6 59

4.2-9 Stacked plot of reduction in subset 7 60

CHAPTER I

INTRODUCTION

1.1 Statement of the Problem

Computer systems and networks have become a voluminous and complex web of data

upon which our daily activities have come to rely Devices such as laptops and cell phones have

found their way onto networks through advances in wireless technology, further expanding the

range and depth of these networks High-speed networks provide immensely greater bandwidth,

allowing large amounts of data communication across these systems The availability of the

Internet has become ingrained as a critical component of the infrastructure in nearly all facets of

society, from work to leisure

These reasons, as realized in the responsibilities of system administrators, have a

compounding and complicating effect on the necessity to monitor and protect computer systems

and networks Intrusion Detection Systems (IDSs) have come to the aid of administrators in this

regard, allowing the automation of network and host analysis IDSs monitor for events by

placing a sensor on a data stream and generate alerts when suspicious activity is detected The

situational awareness (SA) of a sensor, i.e the context by which it can generate alerts, is limited

to the information it can gather from the data stream being processed This limits the big picture

aspect of the alerts that are generated, leaving the problem of correlating alerts from various

sensors in the system to the administrators

A novel concept toward increasing the situational awareness of the system is by gathering

and correlating alerts from numerous sensors at a central location This effort is complicated by

the lack of an accepted standard format in the intrusion detection community for generating

alerts An alert correlation component can preprocess alerts from a sensor and translate the

proprietary formats into a common form consumable by the rest of the correlation framework,

reducing the impact of this issue The complexity of correlating alerts derives from the dynamic

nature of the system, where steps in an attack can be performed in various order Steps may also

occur from different sources or across several days This nature forces correlation components to

keep the context of alerts for some period of time in order to recognize the completion of an

attack when it occurs This imposes severe limitations when the complexity of alert correlation is

further reduced in a strictly sequential processing order

1.2 Importance of the Study

This research focuses on further correlating the output in an effort to reduce the output

This in turn provides convenience to a system administrator, primarily through the employment

of intrusion detection and data fusion centered on the need for enhancing computer security The

amount of effort put into these areas individually is extensive and the combination of these fields

is beginning to take the spotlight as we try to glean information from the variety of data available

Intrusion detection systems have become a valuable tool in determining when an attack

occurs, tracking forensics, and even responding to an attack in order to prevent further

compromise The value of these systems has resulted in sensors being written for each part

within a system, such as hosts, networks, and critical applications In today's large systems, this

results in dispersed sensors generating numerous independent alerts that represent individual

pieces of a larger picture

Data fusion deals with combining data collected from a variety of sensors into a single

common picture that is easier to analyze than the disparate detections of each individual sensor

Through the combination of data, it is possible to further assess the situation and to derive

additional information

Data fusion is being explored by numerous entities around the world In terms of

computer security, it is being applied to intrusion detection in the form of alert correlation These

correlation systems have proven themselves as major advancements with great potential to

provide crucial capabilities to computer security [2] However, the data fusion picture is

incomplete and the alert correlation and data fusion communities sometimes diverge in

methodology

By studying the evolved data fusion models and applying them to the correlation of alerts

in an intrusion detection system, we are able to produce an outlook of the system that can be

refined and lends itself toward providing visualization of that state and improving the situational

awareness of a system administrator

1.3 Objectives

The objective of this research is to propose and implement improvements to intrusion

detection alert correlation Alert correlation is the process of taking alerts from numerous

intrusion detection sensors that monitor networks, hosts, and applications, and combine those

alerts into a single picture with more context than the sum of the pieces By combining alerts

from several different sensors it is possible to detect multi-step attacks from initial login through

the host compromise, reduce false positives, and provide a concise and complete picture of the

state of the system This concept has been explored in several existing studies

The concept of alert correlation closely parallels that of data fusion, which is traditionally

applied in a tactical sense to combine data from various sensors such as radar, sonar, satellite

imagery and many other sources In both cases, the systems have sensors with different strengths

and weaknesses that generate data regarding a specific part of a greater picture This data is

generally useful on its own but limited in context since a single sensor can only monitor a specific

area Through data fusion, it is possible to take the data generated by these sensors and combine

the individual context into a bigger picture This process has the benefit that it is often possible to

determine additional information from these individual pieces by accentuating the strengths and

minimizing the weaknesses of each sensor For this reason, the data fusion concepts should be

the basis of alert correlation research and techniques

In an extensive research effort, researchers at the University of California, Santa Barbara

(UCSB) released an alert correlation concept that seeks to define the process In order to prove

the concept, The Computer Security Group (formerly the Reliable Software Group) at UCSB has

developed an alert correlation system called AlertSTAT that applies the research to correlate

alerts AlertSTAT is built upon an existing intrusion detection framework called STAT, also

available from the Computer Security Group The STAT framework is an extensible system

capable of dynamically loading components that define its behavior and capabilities AlertSTAT

is implemented in an essentially sequential processing order where several components process a

small window of all the alerts in the application This means that of all the alerts in the system,

each component can only utilize a small part of that data because each component must operate in

turn

The alert correlation concept presented is a well thought out and complete model with

extensive correlation capabilities in terms of pulling alerts together into a correlated picture

However, studying this alert correlation concept in light of existing and mature data fusion

models shows that the process refinement level provided in data fusion is lacking in the alert

correlation concept The process refinement level in data fusion provides the critical function of

utilizing information in the system to refine the picture generated by the other levels That is, the

alert correlation concept provides accurate results but does not consider all the available

information to refine those results

The objective of this study is to introduce the concept of process refinement into the

existing alert correlation concept in order to provide a more complete picture of the system

1.4 Contributions

The research contributes an initial look at accounting for process refinement into the alert

correlation concept This is done through the introduction of a look-ahead that is capable of

analyzing alerts in other parts of the system, providing a greater range of context than is currently

available The look-ahead system utilizes this additional information to dynamically adjust its

processing windows to account for additional correlations that analyzes alerts within the system

This capability leads to improved reduction of the overall input stream and minimizes the lag

time for an event to transition from one correlation component to the next

The look-ahead provides advantages over the original static window by dynamically

adjusting the time windows that the correlation components use This dynamic adjustment allows

the system to expand the extents of the time window allowing the capture of additional

correlations when it’s needed and shrink the window to increase throughput when it’s not needed

This combination provides the benefits of reducing the data set through additional correlations

while maintaining early warning and notification of the correlations within the data stream

Furthermore, the dynamic time window calculations save time required to manually adjust static

windows in order to reap the benefits of additional correlation since different static windows must

be set to achieve the optimal correlation Both the ability to achieve further reduction through

additional correlation and generating messages regarding the possible correlations in the data are

important factors for a complete and timely view of the system state

Ultimately these contributions are applicable to the greater picture of introducing events

that have already passed from the system by accessing a database of these alerts and supplying

them to the look-ahead algorithm In this case, the application becomes somewhat easier because

alerts outside of the sequential processing structure can be immediately incorporated as a

correlated alert This application would provide further reduction of the overall dataset with

benefits increasing over time The additional system to manage previous alerts in a database and

reapply them was not implemented but the algorithms presented are directly applicable to that

functionality

1.5 Organization of the Thesis

The remainder of the thesis is organized into four additional chapters Chapter Two

discusses the background and information relevant to understanding the work performed in this

study Chapter Three introduces the work performed and the contributions of that work Chapter

Four presents the experiments, the results of those experiments and the analysis of the results

Chapter Five provides a summary of the conclusion and follows up with potential future ideas for

advancing this area

CHAPTER II

BACKGROUND OF THE STUDY

This study involves the area of computer security with a focus on intrusion detection

systems This is further scoped to the specific area of alert correlation via Data Fusion This

section discusses the background related to this study

2.1 Intrusion Detection

Intrusion Detection Systems (IDS) perform a variety of roles, using a variety of methods,

but all essentially monitor system resources, via a data stream, for malicious or abnormal activity

When this kind of activity is detected, the IDS issues an alert indicating the nature of the detected

event and information surrounding it The component that analyzes the data stream is generally

referred to as a sensor or analyzer

An IDS may employ either anomaly detection or misuse detection in order to determine

what constitutes an event Misuse detection makes use of attack signatures and compares that

against the data stream it is monitoring As such, misuse systems generally have a low rate of

false positives but require diligent signature definitions and constant updates to keep the rate of

false negatives low On the other hand, anomaly detection utilizes historical data about the

system in order to define what constitutes normal use and then seeks deviations from this

definition This system is considerably more dynamic, allowing it to detect previously unknown

attacks and thereby reducing the rate of false negatives However, this system is susceptible to

higher false positives and requires a period of time to learn the normal behavior it should expect

In addition to the method of intrusion detection employed, sensors are also categorized by

the domain in which they operate, the environment, and level of abstraction Some examples of

intrusion detection domains include host, network, and application domains Host-based sensors

run on the monitored host and analyze host data streams such as the system log Network-based

sensors are concerned with network traffic Application-based sensors monitor application

specific details such as an application log, communication between an application and a database

or one of several other examples

2.2 Data Fusion

Data fusion is commonly utilized in a tactical sense to combine the data from numerous

sensors to form a complete picture The most widely used method is the JDL, which has been

studied and revised by several other parties [11,12] This model discusses several levels of data

fusion including Sub-Object Assessment, Object Assessment, Situation Assessment, Impact

Assessment, and Process Refinement Figure 2.2-1 demonstrates the effect of each level on a set

of alerts In this diagram, the individual alerts within the system are represented by the smallest

circles As illustrated by the varying shades in (a), Raw Alerts come into the system from

different sensors in different formats Sub-Object Assessment (b) includes pre-processing the

raw alerts to put them into a common format with a uniform identity Object Assessment (c)

associates the individual alerts into the attacks that they represent, such as pulling all the alerts

related to a port scan together into one object Situation Assessment (d) involves the

determination of the interaction and relations between the entities An example of this would be a

multi-step attack, which requires multiple individual attacks to succeed Impact Assessment (e)

involves the determination or prediction of the impact the attack has, or would have, on the

system including the criticality of the assets involved in the attack Process refinement (f)

includes monitoring and refining the current assessment with additional, whether it be current or

previously determined fusion information or from supplemental databases

8

7

9

14 12

16

11 15

13 10

8

7

9

14 12

16

11 15

6

18

5 3

13 10

8

7

9

14 12

16

11 15

6

18

e) Impact Assessment 10

7

9

6 17

13

14 8

2.3 Alert Correlation

The concept of alert correlation is currently being shaped through the efforts of various

individuals and groups While there is no general consensus on how alert correlation should be

performed, there are some common concepts that have come to the fore-front For example, the

distributed nature of individual sensors requires normalization of alerts to a common format and

attack identification [1.10]

There are also several techniques utilized to perform correlation, the primary ones being

filtering, fusion, and multi-step correlation In filtering, alerts are filtered from the system using

knowledge of the network and the attacks that are detected One example is filtering attacks on

an Apache web-server that attempt to exploit IIS vulnerabilities Another example is an attempt

to exploit a service that is not running on a particular host Filtering requires that the intrusion

detection know something about the system it is monitoring as well as something about the alerts

that are detected Fusion correlates alerts by determining the degree of similarity of the alerts to

some determine criteria For example, two alerts detected by different sensors that are only a few

milliseconds apart, contain the same source and target information, and the same attack type

might be correlated into a single alert The fusion component requires that certain critical

information is available and consistent The multi-step correlation operates through state changes

that move as the pre-conditions of each state are met An example of correlating alerts using this

method is the recon-break-in-escalate model, where an attacker tests the system for

vulnerabilities, uses a discovered vulnerability to break-in to the system, and then finds a way to

obtain access These actions are often abbreviated as r2l (remote to local), and u2r (user to root)

representing the attackers objective in the action In this situation, the detection of recon would

transition the component to check for break-in attempts A subsequent break-in would transition

the component to check for escalation of privilege

2.4 Data Sets

Generating quality data sets has always been a problem for intrusion detection

verification This largely weighs on the need for the data set to not only include attacks, but for

the analyst to understand the attacks and the desired detection of these attacks When this

information is known, it becomes easy to optimize the detection routines for the attacks in the

data set, although this may not always lead to the most accurate detection as attacks vary just as

the background traffic varies

Another issue with collecting data sets is that most significant networks are proprietary

and the organizations that own them are not willing to release this information to the public

2.5 Alert Message Standardization

As mentioned above, the distributed nature of individual sensors and the lack of common

structure beg for a standardized approach The Internet Engineering Task Force (IETF) created

the Intrusion Detection Working Group (IDWG) in order to define a common standard for

exchanging intrusion detection messages This effort builds upon preliminary work as well as the

Common Intrusion Detection Format (CIDF) This working group developed the Intrusion

Detection Message Exchange Format (IDMEF) in order to address the need for a standard

message format However, this working group adjourned without solidifying the IDMEF into an

official RFC It is currently available in an experimental form but has been adopted by a few

intrusion detection systems, including the STAT Framework

In addition to IDMEF, the IDWG also proposed the Intrusion Detection Exchange

protocol that utilizes Tunnels and BEEP channels [9] to exchange messages in a secure manner

2.6 Correlation Engine Framework (STAT)

There are several existing alert correlation systems [4,6,10] at various stages of

development and availability The STAT framework was chosen for this effort due to its

availability and configurability This correlation engine was developed by The Computer

Security Group at the University of California, Santa Barbara

The STAT framework is a highly extensible framework that allows abstract

implementations to be loaded and executed by a core component This requires the construction

of Providers, Extensions, Scenarios, and Responses Providers are modules that provide raw

alerts to the STAT framework Extensions provide the extended events utilized by the specific

scenario prototypes within the system Scenarios define prototypes, which are abstracted rules

that model the state transitions of a successful attack in consuming, non-consuming, and

unwinding transitions Finally, responses can be provided that allow the system to execute some

action based on a state transition This response can be a simple notification to the user to

something more complex such as gathering forensics to support the detected alerts

2.6.1 Data Fusion and Alert Correlation Concepts

Conceptually, the data fusion concept described above must be applied to the concept of

alert correlation, taking into account the specifics of computer security and network configuration

to obtain a higher fidelity description In a series of papers and a book [1,2,3], UCSB has

proposed an alert correlation concept outside the specific reference of data fusion research

However, given the degree and depth that research has gone in the field of data fusion, it is

imperative that this connection be made In fact, the alert correlation concept proposed by UCSB

can be entirely categorized into the revised JDL data fusion model [11] One level that was left

out of this concept is process refinement, which includes the continual refinement of the big

picture generated by the other levels of the data fusion This correlation concept has been

mapped to the revised JDL data fusion levels where applicable in Figure 2.6-1

Sensor Ontology

Pre-Processing

Fill in missing information

Alert Fusion

Combine independent detection of same atack

Alert Verification

Determine the validity

of attack

Thread Reconstruction

Combine alerts of single attack on a single host.

Attack Reconstruction

Factors in based alerts into the thread

network-Focus Recognition

Identify hosts that are the target/source of the attacks

Multi-Step Correlation

Identify attack patterns

Figure 2.6-1 Alert correlation concept associated with data fusion

In the sub-object assessment level, the UCSB alert correlation concept applies the

normalization and pre-processing stages, which are concerned with ensuring a complete

definition of the input alerts provided by individual sensors available within the system Object

assessment encompasses alert fusion, alert verification, thread reconstruction, and attack

recognition as these stages of the UCSB correlation system deal with combining the alerts that

pertain, for the most part, to the construct of a single stage attack, or object Note that alert

verification might utilize information gathered from the system through actively probing the

network as well as consulting a database of the assets and sensor network present in the

monitored system Situation assessment categorizes the focus recognition and the multi-step

correlation since these stages in the alert correlation concept deal primarily with putting together

complex attacks and determining the focus of the attack Impact assessment handles the impact

and prioritization of the impacts within the system This stage might consult knowledge of the

dependencies and criticality of each part of the system The process refinement level might be

loosely qualified by the application of sensor ontology and asset databases but it is largely

missing in the UCSB correlation concept

2.6.2 Correlation Framework

The actual STAT system contains several modules that work together to form the

correlation system It is also of importance to mention that the STAT framework was not built

solely for alert correlation In fact, its roots are in the general sense of intrusion detection where

it would monitor the system log or network traffic It is for this reason that the following

descriptions discuss the general function of each component The specific use of the components

by the AlertSTAT application is explained in section 2.6.3 below

Each of the components of a STAT application contributes a specific role that is briefly

introduced here and described in detail in the subsequent sections The Core provides the

coordination between each of these modules The Extension modules define the event types that

are utilized by the other components Providers are the components that are responsible for

bringing messages into the system by packaging relevant data into the appropriate extension types

and returning them when polled by the core Scenarios are responsible for modeling attacks and

detecting anomalies in the event stream A specific instance of a scenario, which monitors the

data stream for abnormal behavior, is called a prototype Finally, responses are used to execute

some type of action in response to the detection of an event by a scenario

xSTAT

xSTAT is a generic STAT application that provides the ability to quickly create a specific

STAT application through the use of configuration files to load the various components It loads

and executes the extension, provider, scenario, prototype, and response modules per configuration

files These configuration files provide an intricate definition of the interaction of each module

with the system from the prototypes loaded from each scenario to the response associated with

specific transitions within those prototypes

Core

The core provides the general framework that these components operate in by abstracting

management of the global event queue, application time, and other general aspects such as thread

safety Upon startup, the core loads the configured libraries and begins to process the global

event queue This processing cycle is a continuous loop of polling a provider for an event and

then processing the global event queue, delegating each event appropriately according to a

predefined level It is through the use of these defined levels that the current system operates in a

primarily sequential execution order That is, the current level of an event is checked against the

level of each scenario When the levels match, that event is removed from the global event queue

and delegated to the scenario for processing

Extensions

Extensions define the data types that are used in the STAT application These data types

represent the types that providers create and that scenario and response modules use to detect and

being analyzed and often the types that the other modules use to accomplish their respective

functionality

Providers

Providers are the objects that provide events to the core, and thereby to the prototypes

executing within the core In general IDS architecture, these providers are the components that

watch the monitored pieces of the system for events For example, a provider would continually

check the system log for activity and package the messages written to that log into the appropriate

extension format and provide that message to the system for intrusion detection One useful

aspect is that providers can also pull events stored within a saved file to make testing possible

Providers handle the incoming event stream for a STAT application

Scenarios

Scenarios represent the analyzer portion of the STAT framework In the STAT

framework, these scenarios represent the general definition of an attack via a state machine The

scenarios can be auto-generated through the use of STATL, which is a STAT scenario coding

language In general, this is a component that will read STATL syntax and generate C code

implementing the state machine defined within the STATL

Each scenario is instantiated as prototype, which becomes a basic detection unit for an

intrusion The core will cycle events through each prototype defined in the system according to a

predefined level In the case of the alert correlation modules, the level is hard-coded into each

scenario

Responses

Responses represent an action executed when a scenario transitions to a particular state

These can be in-depth responses, representing some kind of intrusion prevention measure, or

simply a notification via a message that the transition represents the correlation and detection of

an attack Since the response is the execution of more code, it is possible to implement complex

responses to perform a variety of tasks Since the STAT family of applications in intended to

work together in a coordinated manner, the majority of the defined STAT applications utilize the

response modules to output alerts in IDMEF

2.6.3 AlertSTAT Framework

The module utilized for this effort is the AlertSTAT application This is an extension of

the xSTAT generic application and utilizes extensions, providers, scenarios and responses

specifically written for this purpose In the case of the alert correlation the extension module

provides types to represent alerts in the Intrusion Detection Message Exchange Format (IDMEF)

version 1.0 These objects abstract the access and manipulation of the data contained in the XML

IDMEF messages

There are two providers that can be utilized with the AlertSTAT application One

provider is intended for the real-time reception of alerts from other STAT applications, or any

application that can generate IDMEF alerts and submit them to a collection component This

real-time provider was not utilized Instead, the second provider was used, which is capable of

parsing a text file containing IDMEF messages and providing one message when polled by the

STAT core This provider is useful for studying and modifying the logic since the set of input

messages can be consistent for each execution

The AlertSTAT application contains several scenarios, some of which are instantiated as

multiple prototypes In particular, the AlertSTAT scenarios utilize three unique types of

algorithms for intrusion detection alert correlation One method, filtering, is simply the filtering

of single alerts based on certain criteria Another method is fusing, which takes two alerts and

compares certain criteria looking for a match When a match is found, these two alerts are fused

into a single alert In the case of the AlertSTAT component, these new alerts are correlation

alerts The final method is the multi-step approach In this approach, alerts cause the transition of

a state machine from one state to another These states can be complex and take the form of

consuming, non-consuming, and unwinding [3]

As discussed previously, each prototype is associated with a specific level that the core

uses to determine whether an event is delegated to a prototype as it iterates over the event queue

The specific order of execution is defined by the circled number in the upper left corner of each

prototype represented in Figure 2.6-2

Scenario

idmef_fuse

Prototype Procpath ProcPathMatcher Local Queue

5

Scenario

idmef_fuse

Prototype Many2One FanInMatcher

Scenario

idmef_fuse

Prototype One2Many FanOutMatcher Local Queue

Local Queue

6 6

Scenario

idmef_multistep

Prototype Multistep

Local Queue

6

Figure 2.6-2 The AlertSTAT system implementation

In the AlertSTAT system, the core contains a global queue that is utilized solely for

events that are transitioning between stages or have just entered the system through the provider

This central queue provides an optimal point to introduce processing into the system that requires

global access or coordination The core removes events from this global queue and delegates

them to each registered prototype for processing The prototype them keeps these event on a

local queue for some defined amount of time This time is referred to as the time window in the

remainder of this work The events that are present on the prototype’s local queue are the only

events that are considered in the correlations calculated by that scenario Once an event falls

outside of the time window, it is removed from the local queue and returned to the global queue

It is of note that the prototypes that use a filter based approach or a strict state machine transition,

do not require maintaining a local queue since they are able to accomplish the roles they are

responsible for instantaneously, as in the case of the filter, or they maintain a separate account of

the alert, such as a state transition Note that a state transition could require multiple events to

occur and therefore, some prototypes based on state transitions require a local queue and time

window

2.6.4 Limitations

One limitation of the current AlertSTAT system is the lack of a process refinement stage

in which the correlations generated by the system, and even the alerts that are not correlated, are

constantly analyzed with the current set of alerts to further refine the picture Some components

that model complex attacks through a state machine requiring single alerts to trigger transitions

accomplish refinement since they remain in the particular state However, the components that

fuse alerts together based on criteria or that require multiple events to transition states lack this

process refinement capability

The time windows that determine the time span within which alerts are considered

against each other for correlation is also static This means that the spatial relationship between

two combinable alerts must be within a specific time period in order for a correlation to occur

The nature of networks is also inherently dynamic, with a multitude of varied traffic flowing even

in the simplest systems In such systems, static criteria can only ever be partially optimal over

time In fact, it is only a matter of time before a malicious user learns that waiting a certain

period of time between attacks provides a way around the system pulling together all the

forensics of the attack Perhaps most importantly, other components are denied the ability to

analyze any alerts for the cumulative time period of all the prototypes configured to execute

before it

An example would be useful toward understanding how the current system currently

correlates alerts using the static window Figure 2.6-3 provides the basis of other examples used

in this description In this example, we are observing a piece of the system where some alerts are

being processed by components earlier in the sequence The timeline that specific events enter

the components in this example are static and defined by Figure 2.6-3 part (a) The desired

optimal answer is represented in Figure 2.6-3 part (b) In this optimal solution, alerts 1 and 3

combine to create alert A1,3, alerts 2 and four combine to create alert A2,4, and alerts A1,3 and 5 combine to form alert BA,5 Alert 6 is identified as an inconsequential alert by the filtering

component and removed from the system This would be an example of either a false alert (false

positive or false negative) or some type of alert that the system is configured to allow, such as a

successful remote login via secure shell Finally, Figure 2.6-3 part (c) represents the correlation

components that are available in the system In this example, there is some part of the system

analyzing alerts with a static time window of two and the solution column represents the timing

that the alerts processed by this system would enter the rest of the system In the part of the

system covered by this example, there are two components that utilize a fusing algorithm and

maintain the events on their local queue for a default 2 time units, as represented by ∆t = 2 In

between the two correlation components is a filter component whose local queue is not utilized

since it only needs to test single alerts for certain criteria The component does essentially have a

local queue, since alerts are removed from the global queue and delegated to the filter component

The filter analyzes the alert data and either returns the alert or removes it from the system right

away That being the case, we represent it with a ∆t = 0, representing the instantaneous nature of

the filter, although the implication of no queue is also correct In this example the subscript of t

represent some consistent elapsed time unit For example, t1 – t0 = 1 unit of time or a ∆t = 1 The time values referenced in these examples use this notation rather than defining a specific unit of

time (e.g 2 minutes or 3 seconds)

b.) Optimal correlation output

Component 1 Fuser A Local Queue ∆t = 2

Component 2 Filter Local Queue ∆t = 0 (No Queue)

Component 3 Fuser B Local Queue ∆t = 2

c.) The components in the correlation system

B A,5

A 1,3

A 4 2

5

A 2,4

B A,5

A 3

Figure 2.6-3 Basis for correlation examples

With the setup referenced in Figure 2.6-3, the following Figure 2.6-4 describes the

sequence of events that occur during the correlation of the original system This illustration

represents one frame of time as a row whose time begins below the horizontal line referencing the

time The columns represent the correlation components in the solution and the contents of the

respective local queues at each time

Figure 2.6-4 Correlation example with static time window

Table 2.6-1 Output from static window example in Figure 2.6-4

Alert 2 (correlation with Alert 4 did not occur) t4

Alert A1,3 (correlation with Alert 5 did not occur) t5

Alert 4 (correlation with Alert 2 did not occur) t7

Alert 5 (correlation with Alert A1,3 did not occur) t8

There are six total alerts in the example in Figure 2.6-4 that are being processed by other

components in the system prior to t0 The entrance of these alerts into the components of interest for this example is defined by their appearance in the leftmost column of the figure The alerts

move into Fuser A at the indicated time and the contents of the local queue at each time ti are represented by the column headed with “Fuser A” In this example, the local queue allows at

most a difference of 2 time units in the queue The same setup applies to Fuser B The Filter

operates on single alerts and does not require keeping the alerts for any period of time After

determining whether the alert passes the filter or not, it is returned to the global queue where it is

delegated to the next component

In this example the time delta between alerts 2 and 4, ∆t = 4, is greater than the static

threshold, ∆t = 2, and the correlation does not occur Also, the time delta between alert A1,3 and alert 5 is greater than the time window utilized by Fuser B Therefore, the output from this part

of the system is alerts 2, 4, 5, and A1,3, which is not the optimal output defined in Figure 2.6-3 part (b) The reduction of the data set is 4 alerts from a total of 6, or about 33%

CHAPTER III

ALERT CORRELATION WITH LOOK-AHEAD

3.1 Correlation Process

The chosen correlation system utilizes many different correlation techniques and

algorithms in order to provide a high degree of accuracy to the correlation process It also utilizes

these techniques in several different modules and interlaces them in an optimal order to improve

the quality of the output The work here strives to maintain these qualities of the original system

while improving the ability of each component to process incoming alerts

The original correlation system operates in a primarily sequential order, as discussed in

section 2.6.3, handing off incoming alerts to the first component as inputs Each component

processes its input alerts, performing its assigned tasks, and then decides whether to output the

alert back to the core for continued processing or to remove it from the system by not returning it

for further processing Once the component outputs the alert back to the core, the alert is

propagated to the next component

The alerts are delegated to the components by the core engine, which manages a central

queue of alerts ordered by time As each alert is delegated, it is removed from the queue until the

correlation component returns it as output Each component is configured to analyze the alerts

with a static time window While an alert is being processed within a single stage of the pipeline,

no other stage is able to process that alert and the flow through each stage of the pipeline is

entirely sequential

The original system performed all of the correlation levels with the exception of process

refinement Once an alert was processed once by each level, the processing on that alert was

complete This was a limiting aspect of the original system and providing the ability to improve

the correlation process is a pivotal concept of taking the current alert correlation systems to the

next level of utility In that frame of mind, the goal of this effort was to improve the ability of the

system to further reduce the dataset and provide an early warning capability, all within the

context of introducing the concept of process refinement into the alert correlation system This

goal was accomplished through the use of a look-ahead system that drives a dynamic time

window used for processing the alerts within the correlation system

3.2 Dynamic Time Window

In the original system, the time windows that the correlation components operate with are

statically defined by initialization parameters It is up to a system administrator to properly

define the time window to adequately continue the propagation of alerts through the system while

allowing for enough spatial context to properly correlate the alerts Since the alerts generated by

the system are dynamic due to the variable nature of networks including the manner and order in

which they are generated, a system administrator would need to continually adjust the window to

maintain optimal performance

To improve the potential of the system to identify the combinable alerts, a dynamic time

window was implemented This time window can be set during the execution of the application

to extend the processing window of each individual component, thereby allowing for the

incorporation of additional alerts in each correlation This enables further reduction of the data

set

The dynamic window has two modes of operation One is to simply set the window and

leave it at that setting until another set command in executed This has the advantage of

providing additional control in the system, should an external controller be made available, such

as an administrator’s console This interface allows manual control of the system and provides

for the ability to fine tune the detection or experiment with the effect of adjusting the window

during execution

The second mode was utilized for this effort as it provides the greatest deal of

automation In this mode, the dynamic window is increased by some time and will shrink back to

the originally configured time after a set timer expires The return to the originally configured

time occurs in synch with the passage of time in the system, so that eventually the time window

returns to the configured processing time but still accounts for the spatial relationships between

the alerts and their appropriate correlation That is, the system is able to extend the time window

so that two specific alerts are processed within the same window, which enables the correlation of

those alerts An exploration of how this capability is utilized to improve the reduction rate of the

system is done in subsequent sections

3.3 Process refinement through a look-ahead

As previously discussed, the sequential pipeline nature of the correlation system has

advantages over a similar system operating in parallel One is the obvious simplicity of the

system, which allows the concepts to be explored and studied in the most predictable manner

More importantly a look-ahead allows the organization of the system to correlate in an efficient

order For example, the preprocessor must process the events first so that they are uniformly

represented in the system A filter would then run through the alerts to remove any activity that is

deemed allowable by the system in order to limit the alerts that are analyzed by the other

complex attacks occurs more efficiently The sequential nature of the system allows for these

benefits

In order to gain additional performance from the system, while maintaining the

efficiencies of sequential processing, a look-ahead module was created that enhances the overall

reduction of the system The look-ahead system itself is described in the sections below

3.3.1 Algorithm

Each correlation component contains the algorithms that perform the actual correlations

The look-ahead utilizes these same algorithms in order to test alerts during the look-ahead phases

The event queue processing provided in the core was modified to send alerts to components

ahead of their scheduled time, hence the term look-ahead When components are provided the

alerts ahead of time, the opportunity to analyze the alert is presented along with a chance to

manipulate the current state of the system This section describes the basic algorithm utilized to

perform the look-ahead operations

An early alert presented to the look-ahead is analyzed against the current local queue of

alerts for correlations If no potential correlations are detected, the look-ahead returns and the

system continues to operate as before If a potential correlation is found, the time window is

extended and an expiration timer is set marking how long the extended time window should

remain in place

Once the look-ahead determines a correlation is possible between an early and an existing

alert, it determines the time delta between the two alerts This time delta is compared first against

the current time window that the correlation component is using If the time delta is greater than

the current window, the time window is set to the time delta and a timer is set to the difference of

the left extent, or later in time, of the time window and the time of the early alert Since the time