Beats Filters

This subsection contains information regarding usage and configuration of filters.

Every system generates a large number of messages and in most cases there is no need to collect everything. Therefore there is an option in Logmanager to filter unnecessary messages. For this purpose Logmanager is using Beats agents native filtering functionality. Filtered messages are discarded locally before they are forwarded to Logmanager, thus saving a little bit of processing power.

There are two basic ways of working with filters:

• Whitelist: Forward onlyselected messages, discard everything else.
• Blacklist: Discard only selected messages, forward everything else.

Both ways are valid and can be used interchangeably, depending on your needs.

Clicking on the plus icon in the right corner will open the new page where you will be able to define new filter.

First you need to give your filter a proper name. Then by using available Blockly blocks, create filtering logic which suit your needs.

If you set both this filter and a global filter, they get both applied.

Here are a few examples:

1. Discard all Windows events where ID equals to 5158.

   

2. Discard all Windows events where ID DOES NOT equal to 5158. Other way to think about this: Forward only events where ID equals to 5158.

   

3. Discard all Windows events where ID equals to 4624 AND field TargetUserName contain “$” char at the end (meaning it’s a machine account).

   

4. Discard all Windows events where field TargetUserName matches regex “mail[1-5]{1}", meaning only logs where TargetUserName field equals mail1, mail2, mail3, mail4 or mail5 will be discarded. Everything else that does not match regex rule, for example, mail10 or exchangemail2 will be forwarded.

   

5. Discard all Windows events where field Channel contains Microsoft-Windows-WMI OR Microsoft-Windows-Push.

   

When using the “discard if” function for filtering, be careful not to put these statements one after the other; it is important to combine conditions using logical operators. For example, if you add separate conditions, the logical operator is “AND”.

If you use grouped conditions, the logic is “OR” - this is best illustrated by the example mentioned below. In it, the logical "=" is used, hence the use of “OR”

Another example of a correctly written filter.

For this example we use the negated values of ≠, and for the combination of event channels of logical operators “AND”. In this filter, all events are discarded, except those generated from the channels mentioned below. In other words, when applying this filter, you will only receive the following logs from the device: Application, Security, System, Microsoft-Windows-Sysmon/Operational.

You can find other examples in the Block > Examples section:

It is possible to filter the latest fields that are in decoded JSON format from raw messages.

You can also inspect how the Blockly logic will look like translated to XML or YAML.

XML or Extensible Markup Language is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. It is used by Logmanager to translate graphical Blockly blocks to machine-readable code.

YAML is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. All configuration of Beats Agents is written using YAML, thus Blockly has to be translated to this format to be able to work with Beats.

To edit a filter click a pencil icon next to the filter name.

To permanently delete a filter, click a X sign next to the filter name then in the pop-up windows click YES.

Filter will not be deleted if it is assigned to any agent.