Beats Filters
This subsection contains information regarding usage and configuration of filters.
Every system generates a large number of messages, and in most cases there is no need to collect everything. Therefore, there is an option in Logmanager to filter unnecessary messages. For this purpose, Logmanager uses Beats agents native filtering functionality. Filtered messages are discarded locally before they are forwarded to Logmanager, thus saving a little bit of processing power.
There are two basic ways of working with filters:
- Whitelist: Forward only selected messages, discard everything else.
- Blacklist: Discard only selected messages, forward everything else.
Both ways are valid and can be used interchangeably, depending on your needs.
Beats filters page
Clicking on the plus icon in the right corner will open a new page, where you can define a new filter.
Adding a new filter
First you need to give your filter a proper name. Then, by using available Blockly blocks, create filtering logic which suits your needs.
If you set both this filter and a global filter, they both get applied.
Here are a few examples:
- Discard all Windows events, where ID equals 5158.
Example 1
- Discard all Windows events, where ID DOES NOT equal 5158. Other way to think about this: Forward only events, where ID equals 5158.
Example 2
- Discard all Windows events, where ID equals 4624, AND field TargetUserName contains “$” character at the end (meaning it’s a machine account).
Example 3
- Discard all Windows events, where field TargetUserName matches regex “mail[1-5]{1}", meaning only logs,
where TargetUserName field equals mail1, mail2, mail3, mail4 or mail5, will be discarded. Everything else, that does not match the regex rule, for example mail10 or exchangemail2, will be forwarded.
Example 4
- Discard all Windows events, where the field Channel contains Microsoft-Windows-WMI OR Microsoft-Windows-Push.
Example 5
When using the “discard if” function for filtering, be careful not to put these statements one after the other; it is important to combine conditions using logical operators. For example, if you add separate conditions, the logical operator is “AND”.
If you use grouped conditions, the logic is “OR” - this is best illustrated by the example mentioned below. In it, the logical "=" is used, hence the use of “OR”
Using grouped conditions (OR)
Another example of a correctly written filter.
Another example of a correctly written filter (OR)
For this example we use the negated values of ≠, and for the combination of event channels of logical operators “AND”. In this filter, all events are discarded, except those generated from the channels mentioned below. In other words, when applying this filter, you will only receive the following logs from the device: Application, Security, System, Microsoft-Windows-Sysmon/Operational.
Using grouped conditions (AND)
You can find other examples in the Block > Examples section:
Filtering examples
It is possible to filter the latest fields, that are in the decoded JSON format from raw messages.
You can also inspect, how the Blockly logic will look like translated to XML or YAML.
XML view
XML or Extensible Markup Language is a markup language, that defines a set of rules for encoding documents in a format, that is both human-readable and machine-readable. It is used by Logmanager to translate graphical Blockly blocks to machine-readable code.
YAML view
YAML is a human-readable data-serialization language. It is commonly used for configuration files and in applications, where data is being stored or transmitted. All configuration of Beats Agents is written using YAML, thus Blockly has to be translated to this format to be able to work with Beats.
To edit a filter, click the pencil icon next to the filter name.
Editing a filter
To permanently delete a filter, click the X sign next to the filter name.
Deleting a filter
Filter will not be deleted, if it is assigned to an agent.