1. Short Description
The MMAS aims to use
state-of-the-art technologies and methods for Thermal and Visible Near-InfraRed
(V-NIR) image processing to detect man malicious activities near an
infrastructure, like detection of moving objects, people or animals during the
night or day. MMAS module is composed of a Bi-spectral camera and a Processing
Unit (server). An operator has access to a User Interface (UI) to configure,
manage alerts and warnings. The MMAS is expected to
assist not substitute the operator, so it will help in the detection of
physical threats in critical infrastructures. The work to be performed by the
MMAS comprises three main features, which are detection, classification and
tracking of targets of potentially physical attacks on a structure protected by
7SHIELD. The MMAS will make available a video with alarms, this video gives
some awareness of surroundings via visualization of specific areas.
2. Main Purpose and Benefits
The safety and security of
infrastructures around the world are at constant risk due to either physical or
cyber-crimes. Security agencies are usually in charge of dealing with these
threats and protect the proper functioning of such facilities. In order to
efficiently survey and protect these areas, video vigilance equipment is common
use.
Visually analysing multiple video
streams captured by the cameras can be a repetitive and boring process that can
lead to missing some relevant events. After only twenty minutes of watching and
evaluating monitor screens, the attention of most guards can go well below
acceptable levels. This decrease in capacity can be up to 95% loss, which
usually leads to missing possible important occurrences. The accuracy of
incident detection can decline by about 10% to 15% after only about 30 min of
watch time and then show a more gradual decline over time. This problem becomes
worse if the guards are looking at multiple displays. When the observers
monitor four displays they can miss about 20% of the targets, a number that
goes up to 60% when the number of displays increases to nine.
The use of computer assisted
detection systems can help to overcome this problem and has been in constant
growth paired with the latest technological advances. This kind of systems are
meant to alert the observers for a possible occurrence, reducing the
possibility of missing an event. The first implementations, included the use of
motion detection cameras and advanced motion detection systems. These solutions
came with the necessity of custom calibrations for each implementation in order
to try to mitigate their significant false positive ratios.
Having into account these
problems, Task 4.5 sought to develop a
Multi-Modal Automated Surveillance (MMAS) system based on the last AI
techniques, that will improve the detection of possible threats, efficiently
and with less human effort or intervention. This system must need minimum human
intervention during the operational time, generate alarms with low percentage
of false positives and be easily integrated into the complex 7SHIELD
system.
As
7SHIELD aims at providing a holistic framework enabled to confront complex
threats by covering all the macro stages of crisis management, specifically
early warning mechanisms that estimates the level of risk before the occurrence
of a cyber or physical attack. The MMAS is an ideal candidate for the detection
of possible physical threats based on imagery/video detection. The MMAS is
located at the first layer (Figure 1), in the sensor level, of the 7SHIELD
hierarchical architecture, it should be deployed in the Ground Station (GS)
facilities.
Figure 2‑1 MMAS within the 7SHIELD
3. Main Functions
The work to be performed by the MMAS comprises three
main features, which are detection, classification and tracking of targets of
potentially physical attacks (night and day) on a structure protected by
7SHIELD. It gives some awareness of surroundings via visualization of specific
areas.
The MMAS module provides the 7SHIELD with three main events or
detections within a FOV of a thermal camera, based on video frames;
- detection
of high level of heat,
- motion
and
- classification
of intrusions with the following classifications
- person,
- animal
(reindeer) and
- vehicles.
3.2 Function 02Transmits all the events or detections to other 7SHIELD Modules through a KAFKA broker.
3.3 Function 03
The MMAS is expected to assist not substitute the camera
operators, so it will help in the detection of physical threats in critical
infrastructures
3.4 Function 04
The MMAS will make available a
video with alarms, this video gives some awareness of surroundings via
visualization of specific areas.
4. Integration with other Tools
The MMAS server is connected to a
7SHIELD LAN with the use of an Ethernet interface or Wi-Fi. A local or remote
broker is used to communicate among 7SHIELD modules, it permits to receive or
send data in an efficient way. The messages generated by the MMAS algorithms
contain alarms, control information and images. The messages
are formatted in JSON (JavaScript Object Notation), which is a lightweight
format for storing and transporting data.
The following information is sent to
7SHIELD:
- Messages of alarms and warnings
- Position of targets (pan, tilt and zoom)
- Images of targets
- Classification of targets
- A URL of a video feed
- Control information to 7SHIELD modules
ecosystem
The MMAS module provides the 7SHIELD with three main events or
detections within a FOV of a thermal camera, based on video frames as described
before;
- detection of high level of heat,
- motion and
- classification of intrusions with the following
classifications
- person,
- animal (reindeer) and
- vehicle.
The three events produce information on three differentiated
topics, namely;
- DL.MMAS.ONE.HEAT (for heat detection messages)
- DL.MMAS.ONE.MOTION (for motion detection messages)
- DL.MMAS.ONE.CLASSIFICATION (for classification
messages)
and status request/response (status messages that transmit the
status of the MMAS) topics for this module are;
- DL.MMAS.ONE.STATUS.RESP
- DL.MMAS.ONE.STATUS
These messages follow the format
IDMEFv2 v0.3. The events are embedded in the 7SHIELD format in the object “alert”.
The attached contents (“Attach”), are images (format jpeg) taken from the
camera with the alerts designed over the entities that were the cause of the
alert. In the attachment the URL of a feed video of the processed video with
alarms, is also transported.
MMAS offers a technical User
Interface (7 User interface),
this interface permits an operator to position the camera over the area of or
areas of interest and to program the alarms of the MMAS.
5. Infrastructure Requirements
The MMAS
is compose by Processing Unit (PU) (server/PC ) with a GPU and a sensor, these
need basically electric power, communication interfaces (to connect to
the 7SHIELD LAN) and user interfaces.
MMAS Processing Unit
The MMAS Processing Unit (PU) is an
application developed to run locally, it integrates a video Web server proxy to
protect the cameras from external access.
The PU is deployed in a server/PC with physical interfaces, for
communications namely;
- Ethernet or Wi-Fi,
- a monitor,
- a keyboard,
- a mouse
and
- power 220 AC
The MMAS Processing
Unit performs all the processing of the MMAS, and needs interfaces for:
- Communication with the Bi-spectral PTZ
cameras.
- Communications with other 7SHIELD
Modules through the existence of a KAFKA broker
- Creates a Web video stream server to
offer video stream images with alarms to other 7SHIELD modules.
- Creates a User Interface (UI) for an
operator.
MMAS Sensor [4]
The sensor selected
for the MMAS is an AXIS Q8742-E Bispectral PTZ Network Camera, composed of two
internal cameras and one PTZ control. The cameras work in the Visible
Near-InfraRed (VNIR) and in the Long Wavelength Infrared (LWIR), aka Thermal,
the camera may work from -40ºC to 55ºC.
The sensor is deployed with
physical interfaces, for communications namely;
- Ethernet (for communication with the
MMAS PU)
- 20-28 V DC,
typical 16W, max 204W
- Dimensions 244
x 360 x 582 mm
(9.5 x 14 x 23 in)
- Weight 15.1 kg (33.3 lb)
- IP66
- Pan: 360° endless, 0.05°–120°/s
- Tilt: -90° to +45°, 0.05°–65°/s
Figure 5‑1
AXIS Q8742-E Bispectral PTZ Network Camera
To control the cameras, an Open API
for software integration (including VAPIX®) is provided and a HTTP-based video
interface providing functionalities for requesting single and multi-part images
and for getting and setting internal parameter values. The image and CGI requests
are handled by the built-in web server. The cameras are connected directly to
MMAS Processing Unit with an Ethernet
cable.
6. Operation Manual
The MMAS is compose by Processing Unit (PU)
(server/PC ) with a GPU and a sensor, these need basically electric
power, communication interfaces (to connect to the 7SHIELD LAN) and user interfaces
(see 5 Infrastructure
Requirements).
As the MMAS is compose by two main
modules, they communicate with each other and with other 7SHIELD modules,
namely with the KAFKA broker. This is accomplished using two possible
configurations, in set-up one (Figure 6‑1), the
MMAS Sensor is connected to the MMAS PU by an Ethernet interface using a direct
cable, in set-up two (Figure 6‑2) the
MMAS Sensor is connected to the Local area Network (LAN), using an Ethernet
interface , this time the MMAS PU reaches the MMAS Sensor through the LAN, in
both situation the MMAS Sensor should be set with a fixed IP in order to be
reached by the MMAS PU. The MMAS PU
accesses the 7SHIELD KAFKA using an Ethernet or a Wi-Fi interface connected to
the LAN , this also permits other 7SHIELD modules to access the video server of
the MMAS PU, for this the MMAS PU IP should be also fixed.
6.1.1 MMAS Processing Unit (PU)
The MMAS PU is a standalone
hardware composed by a personal computer,
with an internal Graphic processing Unit
(GPU) needed to work with real time classification algorithms. The MMAS PU
comes already with the MMAS application installed and configured with ports and
firewalls. The installation of the PU needs a room with, an Ethernet or Wi-Fi
connected to the 7SHILED LAN. User interfaces like a monitor, a keyboard, a
mouse and electrical power 220 AC.
6.1.2 MMAS Sensor
The installation of the MMAS Sensor
requires technical knowledge and should follow the installation manual
procedures referred in the AXIS Q8752-E Bispectral PTZ Network Camera installation
guide [1]. The installation should take in consideration the physical distance
to the 7SHIELD LAN or to the MMAS PU, the size, the dimensions, and weight of the camera. The MMAS Sensor
requires an extra hardware that converts 220AC power supply to 24VDC (see Figure 6‑2 ),
the distance to the camera should not be higher than 20meters. The camera can
be installed in different configurations (Figure 6‑5)
depending on the requirements of the site.
Figure 6‑5: Installation
configurations
Some pictures of the installation
manuals [1][2][3] are shown here (Figure 6‑6) to
be used just has a reference, the installation of the camera without consulting
the installation guide is not advisable.
Figure 6‑6: Screenshots from the installation manual. Soon as the camera is powered on it
will available to be used, if it is connected directly to the MMAS PU or to the
LAN it can be used, no user actions are required after a successful
installation.
The MMAS offers a technical User
Interface (see Figure 7‑1),
this interface permits an operator to position the camera over the area of or
areas of interest and to program the alarms of the MMAS. The MMAS UI is
composed of two main interfaces; the control interface (Figure 7‑1-point
2), which permits to set several parameters and alarms and the video interface
(Figure 7‑1-point
1), which shows the FOV of the camera.
Figure 7‑1 MMAS Technical User
Interface
The control interface (2) is
composed of several controls namely;
- Video control - permits the operator to go
back to the last seconds of the video images produced by the camera, it also
permits to stop the video at the moment of visualization.
- Image Control- with this control the
operator changes the colour pallet, between four colours the, Bone, Winter, Jet
and Raw. In the picture is set the Jet pallet.
- Modes - this widget with two
options, Automatic and Manual, are mutually exclusive and permit the MMAS to
enter in manual or automatic mode. In automatic mode the MMAS disables the UI,
the manual mode permits the operator to change all the settings of the U, the
picture shows the interface in manual mode.
- Detection - is composed of three check
options; Classification, Motion and Level. By checking each of these check
boxes the operator enables the alarms produced by these detections’ methods.
The “Therm Level” scroll widget permits to set the
minimum level of heat that produces an alarm.
- Therm an Visual
Zoom -
permits the operator to change the zoom of the cameras.
The video interface (1) is
basically a window video with superimposed information;
- MODE – this informs the operator on the
mode that the MMAS is working, manual or automatic.
- PALLET – shows all the colours of the pallet that is in use.
- PAN and TILT – informs the operator
about the position of the cameras in terms of tilt and pan.
- ZOOM - Gives the level of zoom of the
image.
- ALARM – indicates the area of the image
that generated the alarms in this case a classification alarm and a thermal
alarm, given by red rectangles.
Figure 7‑2 Thermal images with
classification alarms
Figure 7‑3 is a
screenshot of the VLC application
connected to the MMAS video stream, this stream can also be used by a web
browser or be embedded in an application.
Figure 7‑3 VLC application with MMAS
video stream
In the automatic mode the MMAS will
follow a script in which is described the time that the camera will be set to a
position defined by the values of Pan, Tilt and Zoom, to facilitate the
creation of the script an application was developed, Figure 7‑4 shows
the interface of this application.
Figure
7‑4
Automatic mode application
The user sets the values of Pan and
Tilt in degrees for each time interval, the camera will follow the states one
after the other. For example, in the Figure 7‑4 the
camera will have a fixed position (Pan16 and Tilt 83 degrees) for 20 seconds
and cyclically will change the position (Pan 0 and Tilt 0 degrees) for 10
seconds. The application also permits to
insert masks to the MMAS, which is achieved through the insertion of an area file, each of these
files will include several mask areas, which will be used to mask critical
areas that may generate non-controlled alarms.