1. Short Summary
Alert
correlation is a process required to manage the volume of data produced by the
various Intrusion Detection Systems (IDSs) in the system and to provide an
overview of the status of this system. Indeed, in an information system
generating a large volume of data, the number of security notifications can be
very high. This high number is the result of several phenomena. The same attack
can cause the emission of several alerts of the from one or more IDSs.
Repetitive or sequential attacks (e.g., port scan, network scan, machines,
denial of service, brute force attack) can lead to the generation of several hundreds
of alerts per IDS. It also more important to correlate cyber and physical
alerts to have an overview of behaviour stats of the supervised system.
The Hyper Combined Correlator (HCC) module uses correlation rules to correlate different alerts
from cyber, availability and physical alerts and emit a correlation alert if
predefined scenario matches. A scenario is defined using correlation rules
developed in python programming language or using high level syntax. This
module integrates a secure communication layer. It could connect to the Kafka
broker via authentication certificates. This makes the exchange of messages
between the different topics secure also guarantees the confidentiality of the
data.
The correlation process aims to
merge all alerts involved in the attack detected by the Hyper Combined Correlator (HCC). These alerts are referenced by a CorrID attribute in Unified Alert Format (UAF) generated message.
The HCC is based on correlation
rules. Such that each correlation rule makes it possible to define an attack scenario
and each attack scenario is defined by a context. The Hyper Combined Correlator (HCC) generates a correlation alert after that the duration
expiration is over.
The functional process of the Hyper Combined Correlator (HCC) is divided into 3 parts:
-
Vérification and matching
incoming data
-
Aggregation of incoming alerts
-
Enrichment and correlation of generated
alerts
2. Main Purpose and Benefits
As explained earlier, HCC is an
essential and central building block of 7SHIELD whose purpose is to link all
detection layers.
Each detection module sends
information to the correlation modules. The physical detection modules report
to the Geospatial
Complex Event Processing Engine (GCEP), the cyber
detection modules report to the Cyber-attack Correlator (CAC) and the Availability Detection Monitoring (ADM) reports to the AC. However, the correlations made by these
modules do not make it possible to correlate information by the three
environmental detection zones of the 7Shield system.
The main objective of HCC is
therefore to unify these 3 types of detection and to allow the correlation of
all events wherever they come from.
3. Main Functions3.1 Correlator Alert Sending
The main function of HCC is to
allow the issuance of correlated rules created from other alerts.
Unlike Geospatial Complex Event Processing Engine (GCEP), AC, and Cyber-attack Correlator (CAC), HCC allows any
alerts in the system to be correlated with each other. It can therefore correlate physical alerts,
cyber alerts, availability alerts but also correlated alerts.
3.2 Create Correlation Rules
It is important in this type of
module to allow full adaptation to the target system. Therefore, HCC allows the
creation of correlation rules. A correlation rule is the element that dictates
the functioning of HCC. It defines what is needed to create the rules. Although
some standard rules common to any system may exist, it is important to allow
for the enrichment and adaptation of the rules issued over time.
The HCC is a module of the
situational Picture Layer of the 7SHIELD architecture. It allows physical,
cyber and availability alerts from the Geospatial Complex Event Processing
Engine (GCEP), AC and Cyber-attack
Correlator (CAC) modules to be correlated.
This tool does not have a user interface. It only requires the
writing of correlation rules by the user to issue these correlated alerts.
To
perform these correlations, it requires listening to the necessary Kafka broker
topics. It must also be able to send the newly correlated alerts to the Kafka
topic provided for this purpose.
HCC, therefore, takes the following data as input:
-
Physical alerts from the GCEP
module
-
Availability alerts from the AC
module
-
Cyber alerts from the CAC
module
The HCC, therefore, issues the correlated alerts in the Kafka exit topic.
5. Infrastructure Requirements
The
installed HCC only represents a docker container. It can therefore be installed
on any machine with the following dependencies:
The HCC
can be installed on premise or remotely. The only requirement for its operation
is to be able to reach and write to the Kafka broker.
6. Operation Manual
6.1 Set-up
6.1.1 Install and start the HCC
Prerequisites:
·
To have cloned the HCC git
repository
·
To have a folder that contains
all needed certificate for the broker connection
·
Have configured the HCC (as
described below in the section "Configure the HCC")
First, we need to configure the
global environment variables for the execution of the docker-compose main
command. To do this, simply edit the .env file.
The following variables must be
filled in:
-
CURRENT_ENV: The path to the
env file used for environment variables internal to the container
-
CURRENT_RULES: The path to the
folder containing used correlation rules
-
CURRENT_CERTIFICATES: The path
to the folder containing all certificates files used for Kafka connection.
The only operation required to
install and start the HCC is to run the following command in the root of the
HCC repository.
docker-compose up -d –build
To avoid rebuilding the HCC docker
image each time, it is more appropriate to run the command without the build
argument, i.e:
docker-compose up -d
To access the HCC logs, use the
following command:
docker-compose logs -f
6.1.2 Main structure of the HCC repository
The git repository contains the
following main folders and files:
-
env: This folder contains
environments files used to build the environment of the HCC docker container.
-
files: This folder contains all
the files useful to the container.
-
rules: This folder contains the
correlation rules used by the HCC.
-
tests: This folder contains a
python script used to inject alerts in the topics consumed by the HCC. This is
used to test the triggering of a correlation rule. The folder also contains the
test alerts that will be injected.
-
.env: This
file contains the main environment variables needed to run the docker-compose
command.
-
correlator.Dockerfile:
This file is the Dockerfile used to build the docker image of the correlator.
-
docker-compose.yml:
This file is the docker-compose file used by the docker-compose command.
6.1.3 Configure the HCC
Set up the container
environment:
The env folder contains multiple
environment files which describe the useful environment variables for each
environment.
The naming convention used is
<ENVIRONMENT NAME>.env.
The following environment variables
must be filled in:
-
COMPOSE_PROJECT_NAME=HCC: Keep
its value at HCC
-
KAFKA_SERVER: Contains the
complete url (hostname:port) of the Kafka Broker
-
ALERTS_TOPIC: Contains the list of all topics consumed by
the correlator. Each topic must be separated by a space.
-
CORRELATION_ALERTS_TOPIC:
Contains the topic on which correlated alerts will be sent
-
KAFKA_CAFILE:
The container internal path to the CA file used for the Kafka connection.
-
KAFKA_CERTFILE:
The container internal path to the cert file used for the Kafka connection.
-
KAFKA_KEYFILE:
The container internal path to the key file used for the Kafka connection.
-
KAFKA_PASSWORD:
The certificate password used for the Kafka connection.
Set up
the rules folder:
To set up the
rules folder, simply put all the correlation rules you want to use in one
folder (in the rules main folder). The path to this folder should be entered in
the main .env file.
Set up
the certificates folder:
To set up the
certificate folder, simply put the certificates and other files needed for
authentication when connecting to the broker in the same folder. The path to
this folder should be entered in the main .env file.
6.1.4 Configure the environment The only thing to configure is to
ensure that the correlator can reach the Kafka broker.
6.2.1 General working of the HCC
This section presents the
functional behaviour of Hyper Combined Correlator. HCC communicates directly
with the Kafka Broker by consuming UAF alerts from input topics and sending
generated alerts in the output topic.
Here are explanations of the
different steps:
1.
HCC receives UAF alerts coming from different
Kafka Broker topics.
2.
HCC correlates incoming alerts
based on the provided ruleset.
3.
Upon successful correlation.
UAF correlation alerts are raised and sent to the Kafka Broker in the output
topic.
Figure 6‑1 General working of the HCC
A few explanations on HCC
processing in accordance with the figure above:
1.
HCC receives UAF alerts coming
from Kafka Broker topics.
2.
HCC transmits alerts to the
correlation rules. If an alert matches an existing rule, that means the alert
is liable to cause a correlation alert. Otherwise, there is nothing to do.
3.
If the context matching the
alert does not exist, some UAF fields are filled.
4.
When the context expires or if
the correlation rule has an ALERT.alert() statement, a correlation alert is
raised, if the threshold has been reached, otherwise the context is destroyed.
5.
A correlation alert is sent to
the Kafka Broker.
6.2.2 Writing a correlation rule
HCC brings a Python correlation
engine. This engine simply executes the requested Python code by providing
different helper classes to be properly integrated into the correlation engine.
Each correlation rule must be saved
in the folder previously filled in the .env file.
A rule can be added by creating a
new python file in this directory. This file must contain a Python class with
the same name as the file (except for the py extension).
The last function to implement is
run(), it takes a single argument: the current UAF alert received by HCC.
6.2.2.1 Importing modules
Some Python modules are required at
runtime for correlation rules. They must be imported from plugin files.
Python provides different ways to
import modules. You can use the import statement or the from statement.
-
import X imports the module X and creates a reference to that module in the
current namespace. After you have run this statement, you can use X.name to
refer to things defined in module X.
-
from X import a, b imports the module X, and creates references in the current
namespace to the given objects. Or in other words, you can now use a and b in
your program.
The following table shows the
different modules used in existing rules with an explanation for each of them.
|
Import
|
Meaning
|
|
from
preludecorrelator.pluginmanager import Plugin
|
Import the “Plugin” class from the
“pluginmanager” module
|
|
from preludecorrelator.context
import Context
|
Import the “Context” class from
the “context” module
|
|
from preludecorrelator import
context
|
Import the “context” module
|
|
from preludecorrelator.idmef
import IDMEF
|
Import the “IDMEF” class from the
“idmef” module
|
|
from preludecorrelator.context
import Timer
|
Import the “Timer” class from the
“context” module
|
|
from preludecorrelator import
require
|
Import “require” module
|
Notes:
- The module from preludecorrelator.pluginmanager
import Plugin is imported in each plugin file
- Each python module of the standard library
can be imported into the correlation rules. Any other module will have to be
installed in the Docker container in order to use it.
6.2.2.2 The run() method
Each python rule consists of a
primary python class whose run() method is executed at the reception of an UAF
event.
Explanation:
-
An module import: ‘from
preludecorrelator.pluginmanager import Plugin’;
-
A class inheriting from the
Plugin class;
-
A run() method with a single
argument: ‘uaf_alert.
The argument of the run() method is
the UAF message received by HCC. This message object provides a certain number
of methods.
6.2.2.3 The get() method
The get() method allows to retrieve
a given field from the UAF message using an UAF field path. For instance, to
see if an incoming alert has the category Availability.DDoS, we can retrieve
this value with the get() method.
In the
previous example, categories contain a list of all the categories in the alert.
This is possible because of the "*" character.
As this method
is permissive, it is possible to chain multivalued fields to retrieve more
data.
Example:
This will retrieve the values of
all field2 contains in field1.
6.2.2.4 The match() method
The match() method is similar to
the get() method, except it allows to match a value against a regular
expression:
In the example above, if the value of the alert.field_to_compare UAF
field matches the regular expression, the print() function is called.
Note: Every time that the match() method is
used, you must import the python module ‘re’
6.2.2.5 The set() method
The set() method allows you to set
a value to an UAF path within an UAF object.
Note: The PreCorIDMEF class must be imported to create an UAF object
6.2.2.6 The alert() method
The alert() method allows sending
an UAF object to the Kafka Broker. This is mostly useful when generating a
correlation alert. Here we will make a simple demonstration of how to use these
methods.
An alert is sent containing an UAF
path which is alert.field_to_fill set to ‘Value’.
6.2.2.7. Phyton context
A Context class is available, that
greatly eases the writing of python correlation rules. A context is a named
variable, which you can use to keep track of a given rule state.
The Context class might take
several arguments.
-
name: The context name, used to
identify the context.
-
options:
o
An optional expiration timer
(the context is destroyed on timer expiration)
o
An optional threshold
o
An optional variable which sends
an alert on request.
-
overwrite: Used to erase a
context if the context exists. Possible values are True or False. By default
the value is set to True.
-
update: Allows to load an
existing context. Possible values are True or False. By default the value is
set to False
-
idmef: An UAF message that will
be embedded to the context.
Create
a context:
Update
a context:
Allows a context to be updated.
Creates the context if it does not exist or resets the expiration timer if it
does.
Erase a context:
The context options:
Additionally, several context
creation options might be specified.
- expire: Number of seconds after
which the context should be destroyed.
- alert_on_expire: If a context
expires and this option is set, the embedded IDMEF message will be sent. The
permitted values are True or False. By default the value is set to False.
- threshold: Number of
alerts needed to trigger the emission of the correlation alert. By default the
value is set to 1.
Embedded
UAF object:
Embeds an UAF object to the
context. When the correlation alert will be generated, it will contain all UAF
objects previously embedded to the context.
The
search() method
The search() method allows to look
for an existing context.
-
name: The context name, used to
identify the context
-
idmef: An embedded UAF message,
whose date is used to retrieve the context. No UAF object by default
-
update: Update an
existing context. Possible values are True or False. By default the value is
set to False.
Retrieving a context by the
search() method:
6.2.2.8 Classic rule structure
Here is a
classic rule structure:
The inputs expected from the users
are the correlation rules that must be written and integrated according to the
details previously described in the previous parts.
6.3.4 User Output
HCC only
outputs are the correlated alerts. These are not outputs to the user directly
and the alerts will be visible in the user GUIs of the alert management and
remediation modules.