building collector plugins admin guide

building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide building collector plugins admin guide

Building Collector Plugins

Admin Guide

Copyright © Alienvault 2010

All rights reserved No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any

information storage or retrieval system, without the prior written permission of the

copyright owner and publisher

Any trademarks referenced herein are the property of their respective holders

Table of Content

1 Overview 4

1.1 OSSIM Agent Role 4

1.1.1 Event Collection 4

1.1.2 Event Normalization 4

1.2 OSSIM Server Role 6

1.2.1 Event Enrichment 6

1.2.2 Policies and Actions 7

1.3 The Configuration Workflow 8

2 Configuring Detector Plugins 10

2.1 Rsyslog 10

2.1.1 Configuration File 10

2.1.2 Listener Configuration 10

2.1.3 Filters 10

2.2 OSSIM Agent Configuration 11

2.2.1 Configuration File 11

2.2.2 Parameters 11

2.3 Detector Plugin Configuration 13

2.3.1 Configuration Files 13

2.3.2 Common Event Types 13

2.3.3 Parameters 13

2.3.4 Using Local (Plugin) Variables 15

2.3.5 Using Global (Agent) Variables 15

2.4 Aliases 16

2.4.1 Path 16

2.4.2 Predefined Regular Expressions 16

2.5 Functions 16

2.5.1 Path 16

2.5.2 Conversions 16

2.5.3 Application Specific Translations 17

2.5.4 User Defined Translations 17

2.6 Event Fields 18

2.7 Rules 19

2.7.1 Evaluation Order 19

2.7.2 Structure 19

2.8 Loading Plugins 21

2.8.1 Priority and Reliability values 21

2.8.2 SQL Statement 21

2.9 Plugin Activation 22

2.9.1 Activate the Plugin on the Server Side 22

2.9.2 Activate the Plugin on the Agent Side 22

3 Log files 22

4 Debugging 22

5 Appendix 23

5.1 Regular Expressions 23

5.2 Configuration Example 25

5.2.1 Scenario 25

5.2.2 Write a script to monitor the “last” status 25

5.2.3 Log sample 25

5.2.4 Collect the logs in a new log file 25

5.2.5 Restart “rsyslog” 26

5.2.6 Check whether the new entries are written in the new log file 26

5.2.7 Create a plugin file 26

5.2.8 Register the Plugin with the OSSIM Agent 29

5.2.9 Register the Plugin with the OSSIM Server 30

5.2.10 Check whether the plugin was successfully registered 31

5.2.11 Restart the OSSIM Server 31

5.2.12 Restart the OSSIM Agent 31

5.2.13 Check whether Events and Alarms are received 32

Before starting to write a plugin some actions to reduce the amount of events could be considered:

- Manage the log level settings at the application and managed device level

- Fix the problem that is generating events

- Use Pcap filters to ignore certain hosts or networks (Snort, Tcpdump )

- In deployments with a big amount of analysed data, filtering at the application level should be done whenever possible

o Log Files

Good practice is to use one log file per plugin in order to increase performance Having just one generic log file, all the plugins would have to read the same extensive content in order

to catch the few relevant entries

Using rsyslog it is possible to filter the collected logs based on the syslog tags

Example:

dmz01:/var/log/auth.log:

May 30 13:15:52 dmz01 sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2

o Normalized Event

There is a certain set of fields which are required in order to ensure a consistent evaluation and correlation of the events by the OSSIM server These fields can be populated with information from the log message or statically through the plug-in

1.2 OSSIM Server Role

ossim:/var/log/ossim/server.log:

2010-05-30 06:48:41 OSSIM-Message: Event received: event id="0" alarm="0"

type="detector" fdate="2010-05-30 13:15:52" date="1275239752" tzone="0"

plugin_id="4003" plugin_sid="7" src_ip="192.168.178.20" src_port="4445"

dst_ip="192.168.178.200" dst_port="22" sensor="192.168.178.201" interface="eth0"

protocol="TCP" asset_src="2" asset_dst="2" log="May 30 13:15:52 dmz01

sshd[12980]: Accepted password for root from 192.168.178.20 port 4445 ssh2" username="root"

A Unix server running Samba gets attacked by the Sasser worm Apart from the fact that the attack won’t have an impact on the given environment, it has the potential to exploit a big security hole and for that reason the priority is considered as being high

o Reliability

Classical risk-assessment would refer it as "probability " Since it's quite difficult to determine how probable it is for a network to be exposed to certain vulnerabilities, the IDS related “reliability” approach was considered more appropriate

Range: 0 - 10 Default value : 1

Example:

If a host connects to 5 different hosts in the same subnet using port 445, could be a

suspicious, with 500 connections to different hosts in less than an hour the attack would get more and more reliable

o Asset Value

It is assigned to both the Source and the Destination Hosts and represents the importance the host has to the enterprise

Range: 0 - 5 Default value: 1 (also used for hosts not being defined in the asset database) Example:

A database server can have an asset value of 5, a development test server an asset value of 2 and an unknown host in the Internet causing a portscan event would just have an asset value of 1

o Alarm

Based on the Event Priority (0-5), Event Reliability (0-10) and the Asset Value (0-5), a Risk Value (0-10) is calculated and for values equal or greater than 1 Alerts are generated The Risk is calculated based on the following formula:

Risk = (Priority * Reliability * Asset) / 25

1.2.2 Policies and Actions

Policies are defined in order to define what has to be done with the events as they reach the OSSIM Server:

Correlation (i.e checked against the correlation directives) Forwarding (i.e one copy is sent to the forensic storage) Actions (i.e send an email)

Discard - the last filter possibility before saving the event in the database, although it is recommended to filter the events as close to the source as possible

Policies can make decisions on which events are going to be filtered based on:

Source and Destination Assets (Hosts, Networks, ANY )

Ports

Plugin Group

Time Range

1.3 The Configuration Workflow

o Collect a Log Sample

First thing to start with is checking which log messages the application generates and eventually identify sets of logs having a similar structure Those logs having a similar structure will be where possible covered by a single collector rule

o Create a Plugin File

Best is to copy one existing file and modify its content to match the new application Should a plugin exist for a similar application, it is recommended to copy such a file, as there is a good chance that rules have a similar content and are grouped in a similar way - a generic HTTP-Proxy log will always contain a URL, a generic Firewall log will contain a Source IP Address and Source Port as well as a Destination IP Address and Destination Port Some user defined fields might be defined for a specific application and the correlation at the server level can be simplified if similar applications use the same user defined fields

o Define a Generic Rule

This is the last Rule to evaluate, which catches all the events that cannot be grouped under specific rules

o Define Specific Rules

The Specific rules are defined for specific error conditions or categories of events There might also

be that one single rule is used to generate different types or subtypes of events

o Discard Noise

Events that are considered noise can be discarded by OSSIM by excluding certain event subtypes (Plugin_SIDs) in the plugin file, by the way the regular expressions are defined or by using policies However, the best way to discard events is by filtering them on the monitored device or at syslog level on the host running the OSSIM Agent

o Review the Evaluation Order

The rules are evaluated alphabetically, which means that all it counts is the name of a rule and not the position in the plug-in file The Generic Rule might even be on the first position if the name is properly chosen Having rules alphabetically placed after the Generic Rule will have as effect that the corresponding logs will be evaluated as generic events instead of having the proper event type and subtype assigned

o Register the Plugin with the OSSIM Agent

In order to have a Plugin activated and sending events to the OSSIM server, the path to the plugin file has to be specified in the Agent configuration file

o Register the Plugin with the OSSIM Server

This is required in order to let the server know which events should be expected and which priority and reliability values the events should get assigned

o Activate the Plugin on the Server Side

Restart the OSSIM Server process

o Activate the Plugin on the Agent Side

Restart the OSSIM Agent process

o Testing

Using the logger command sample logs can be replayed in order to test the operation of the OSSIM Agent or Server

2 Configuring Detector Plugins

2.1 Rsyslog

Rsyslog is the Syslog implementation shipped with OSSIM and allows configuring filtering and

forwarding in a really easy way compared to the classical syslog daemon Syslog is also the common method to send and receive logs

Before starting with the plugin configuration it is recommended to check whether the subset of logs the plugin will normalize are saved in an individual file and whether noise can be filtered before reaching the plugin rules

Forward certain events to a local file

if $msg contains 'error' then /var/log/error

if $syslogfacility-text == 'local0' and $msg startswith 'DEVNAME' and ($msg contains 'error1' or

$msg contains 'error0') then /var/log/somelog

Stop processing some events

if $msg contains 'error' then ~

Regex in Rsyslog

http://www.rsyslog.com/user-regex.php

2.2 OSSIM Agent Configuration

2.2.1 Configuration File

/etc/ossim/agent/config.cfg

2.2.2 Parameters

[daemon]

pid: Path to the PID file (Process identifier) [event-consolidation]

Enables event consolidation at agent level It is recommended to use polices instead of this feature as consolidation at the agent level affects the correlation process

by_plugin: List of plugins that will be consolidated enable: Enable or disable (True or False) time: Wait n seconds to consolidate the events before sending them Example:

[event-consolidation]

by_plugin=1001-1150,1501-1550,4001-4010 enable=False

time=10 [log]

Configures the verbose level and the path to the different log files error: File in which the error events will be stored file: File in which all the agent logs will be stored stats: File in which the agent stats will be stored (Every 5 minutes) verbose: Configures the verbose level (Debug, Info, Warning, Error or

Critical) [output-plain]

Writes in a log file what is being sent to the OSSIM Server (Useful for debugging and developing purposes)

enable: Enable or disable (True or False) file: File in which the output-plain will be stored [output-server]

Configures the server to which events are sent enable: Enable or disable sending events to the server (True or False)

port: Listening port of the OSSIM Server

sensor=192.168.178.201 [plugins]

Defines which plugins (detectors and monitors) are enabled name_of_the_plugin=path_to_the_plugin_config_file Example:

[plugins]

postfix=/etc/ossim/agent/plugins/postfix.cfg ssh=/etc/ossim/agent/plugins/ssh.cfg [watchdog]

Monitor the process associated to each plugin (In case it is running in the same machine) enable: Enable or disable (True or False)

interval: Wait X seconds between checks restart_interval: Restart the process every X seconds (This has to be enabled in each

plugin)

2.3 Detector Plugin Configuration

2.3.1 Configuration Files

/etc/ossim/agent/plugins/*.cfg

2.3.2 Common Event Types

Copy and modify the existing plugin files to create plugins of the following types

a. Log - Reading from files

Plugin statement: source=log

b. Database - Reading from databases

Plugin statement: source=database

− mssql - Microsoft SQL

Plugin statement: source_type=mssql

− mysql - MySQL

Plugin statement: source_type=mysql

c. SDEE - Cisco device logs

Plugin statement: source=sdee

d. SnortLog - Snort logs

Plugin statement: source=snortlog

e. WMI - Windows Management Instrumentation

Plugin statement: source=wmi

2.3.3 Parameters

[DEFAULT]

Any variable defined inside this category will be sent to the OSSIM Server if not modified by

a plugin rule User reserved range is between 9000 and 10000

plugin_id: Numerical identifier of the plugin within the OSSIM system Example:

plugin_id=4003

stop: Stop the process when the agent stops (yes/no) startup: Command that starts the process

shutdown: Command that stops the process exclude_sids=SID List Use this option to exclude SIDs Example (hp-eva):

process=snmptrapd start=yes

stop=yes startup=/etc/init.d/snmpd start shutdown=/etc/init.d/snmpd stop exclude_sids=404,200,403

[Rule IDs – Specific Rules]

Here are the events collected and normalized

event_type=event regexp=Regular Expression plugin_sid=Plugin SID Event_Field=Value Example(ssh):

[01 - Failed password]

event_type=event regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*?Failed password for inval user (?P<user>\S+)\s+from\s+.*?(?P<src>\IPV4).*?port\s+(?P<sport>\PORT)"

plugin_sid=1

src_ip={$src}

dst_ip={resolv($sensor)}

src_port={$sport}

username={$user} [Rule IDs – Specific Rules]

[Rule ID – Generic Rule]

Example (ssh):

[99 - Generic rule]

# Nov 15 11:55:35 11.1.4.9 sshd[1769702]: **********

event_type=event regexp="(\SYSLOG_DATE)\s+(?P<sensor>[^\s]*).*?ssh.*"

plugin_sid=99 date={normalize_date($1)}

dst_ip={resolv($sensor)}

Note: As rules are ordered alphabetically the Generic Rule has to have the highest Rule ID

2.3.4 Using Local (Plugin) Variables

The different configuration variables defined in the plugin configuration file can be used with the following syntax:

%()s Example:

process=pads shutdown=killall -9 %(process)s

2.3.5 Using Global (Agent) Variables

\_CFG() Example:

In the agent configuration file (/etc/ossim/agent/config.cfg):

[watchdog]

restart_interval=3600 ; seconds between plugin process restart

In the plugin configuration file(/etc/ossim/agent/plugins/*.cfg):

restart_interval=\_CFG(watchdog,restart_interval)

2.4 Aliases

2.4.1 Path

/etc/ossim/agent/aliases.cfg

2.4.2 Predefined Regular Expressions

The predefined regular expressions can be used when creating new plugins

resolv(host): translates a host name to an IPv4 address

resolv_ip(addr): translates an IPv4 address to a host name

resolv_port(port): translate a port name into its number

normalize_date(date): convert date strings to isoformat (must tag the regular expressions

with the following: <year>, <month>, <minute>, <hour>, <minute>,

<second> or <timestamp> for timestamps To define new date formats add a new regexp to the DATE_REGEXPS array

normalize_protocol(proto): translates the protocols to the protocol numbers, based on the

PROTO_TABLE md5sum(datastring): calculates the md5 checksum

hextoint(string): get the integer value of a hexadecimal number