Beats agents

This subsection explains how to configure and find information about all endpoints (workstations or servers) connected to Logmanager via Orchestrator. The endpoints are added to this table automatically after Orchestrator installation.

This page describes the GUI configuration options. If you are looking for specific information regarding Logmanager Orchestrator agent, such as architecture, installation instructions, changing registry values, diagnostics etc. please go to Logmanager Orchestrator.

This table shows the most important information about connected endpoints. Name of the endpoint (Name), type of the operating system (OS Type), version of the installed Logmanager Orchestrator (Version), IP address (IP), applied filters (Filter – Winlogbeat/Filebeat), number of logs files being collected by filebeat (Log Files), status of Winlogbeat and Filebeat (Status), tags which are associated with the agent (Tags) and last heartbeat from Orchestrator agent (Last Check At).

At the top of the page there is a field which allows searching in all the columns in a table. For example, if you want to search for an endpoint with IP address 192.168.1.1, start typing this IP into the field and the table will be gradually reduced.

To configure the chosen Orchestrator Agent, click on the blue pencil icon. A new page will be opened where you will find following:

• ID: Server-generated unique identifier of Orchestrator - this ID is used for further API communication.
• Hostname: Name of the station.
• Last checked at: Date of last heartbeat. Orchestrator connects to Logmanager at a default intervals (every hour) - so called heartbeats. If there has been any changes to the configuration, issued between heartbeats, Orchestrator will download and apply them to Beats Agents.
• Node ID: Client-generated unique identifier of the Orchestrator.
• OS Type: The operating system where the Orchestrator is installed.

  Currently the only supported OS is Windows.

• OS Version: Version of the endpoints operating system.
• Version: Logmanager Orchestrator version.

  The Logmanager Orchestrator service communicates with the Logmanager at a time interval (heartbeat) of 1 hour. If a configuration change occurs during this interval, the agent automatically downloads and applies it.

• IP: IP address of the endpoint.
• Event Sources: Event Sources configuration.
• Log Files: List of collected logs files.
• Tags: Tags associated with the agent.

This option is relevant only for Winlogbeat agent and enables pulling Windows Event channels from the OS:

Event sources:

• Inherited from Beats Global Config: Use global setting - beatsglobalconfig

  This is a default setting.

• All event sources: Pull all Windows Events (Event Viewer > Windows Logs folder and Event Viewer > Application and Services Logs folder)

  Choosing this option will make the agent send every event generated on the endpoint. Consider applying a filter to limit the amount of collected events.

• System event sources: Pull only System Events (Event Viewer > Windows Logs folder only).

  Winlogbeat will ignore events older than 15min during first and subsequent starts. This can result in loss of data. For example, if you disable Winlogbeat service and leave it in that state for more than 15min, logs older than this interval will not be forwarded.

This option is relevant only for the Filebeat agent. It enables pulling chosen files from the OS:

• Template: Pre-configured paths to commonly collected log files: DHCP, DNS, Exchange, Firewall and IIS.
• File path: Absolute path to a file. Wildcards are allowed, for example applying path: C:\TestData\Log*.txt will instruct agent to pull every file starting with "Log" and ending with .txt in TestData folder and send it to the Logmanager. Each new record (log line) added to the file will be automatically forwarded by the agent.

Take special care with filenames - when creating a new file, Windows by default will hide its type from file name. For example test.log.txt will be presented as test.log in File Explorer. Using the second name as File Path will not yield any results as it technically does not exist, so filebeat won’t be able to find it.

Tags can only contain comma separated alphanumeric characters! You can not use the dash symbol.

Double check, that applied path only select log files, nothing else. Pay extra attention to DHCP logs in Microsoft, the correct path (by default) is: C:\Windows\System32\dhcp\Dhcp*.log. In some scenarios Microsoft stores database data in to the files ending with ".log". When Agent try to pull it, DHCP service on the server might stop working.

Using Wildcard (*) in file path will make it case-sensitive! Reason is full-paths (for example: c:\tmp\test.log) are resolved directly via Windows file-system which by default is case-insensitive, while glob-paths (for example: c:\tmp\*.log) are resolved via Go which makes it case-sensitive.

Before configuring, it is recommended to check that there are not more than 2000 files in a given directory that will be monitored by Filebeat. If there are more than 2000 files in the directory, it is recommended to back up the files or move them to another directory to limit the number of files monitored to less than 2000. Monitoring more than 2000 files can cause various operational or performance problems.

• Tags: String-based tags assigned to each log-line pulled from the file. You can provide multiple tags by separating them with comma. Make sure to use tags which make sense - you will need them to create classification.
• Type: Type of an input file.

Filebeat agent enables usage of many different input types. Most relevant is the "text" type which simply enables pulling any text-based file. This input type is the only one currently supported, but this might change in future releases.

Log Files Tags are important for correct parsing! For an example, if you choose Exchange from template to collect Exchange transport logs, tag "exchange" will be automatically added. Logmanager will use this tag to classify messasges properly meaning direct them to correct parser. If you delete it, your data will not be parsed correctly.

To avoid overloading your Logmanager instance, only new lines from file will be forwarded to Logmanager. Current default filebeat config sets offset at the end of the file. This can potentially result in few log lines being missed during log rotation - when new rotating log file is created application starts writing logs to it before filebeat has a chance to mark the offset.

You can use this option to attach any tag (or tags, as you can add multiple) to logs coming from your Orchestrator agent, for example to specifically mark your endpoint. This option is global, meaning it adds tags to logs coming from both beat agents (Winlogbeat/Filebeat), regardless of tag set in Log Files option.

You can add new tag or select one from drop-down list.

• Version: current version of the agent.
• State: current state of service (running/stopped).
• Target state: expected state of the agent service after next config pull (Enabled/Disabled/Auto).
• Filter: filter config assigned to agents.

If you set both this filter and a global filter, they get both applied.

Filebeat state will only be changed to running if Log Files option is configured. Without it Filebeat agent will not start as it doesn’t have any source of logs to forward.

To delete Beats Agent, click on the red cross. Popup window appears where you can confirm or cancel the deletion.

Deleting agent from this list does not removes it from the endpoint. Make sure to perform agent uninstallation process on the endpoint first.

On older Logmanager it is necessary to manually update classification in order to enable automatic classification for the logs from Logmanager Beats Agents.

Default classification for Logmanagers shipped from factory prior 1st of June 2020 does not 100% reflect new classifier x classifier template logic introduced in Logmanager software 3.5.0. Please, check Your classifier rules for presence of this exact as a last rule (in alphabetic order).

Your classification probably looks like this. In this example there is a default syslog classifier which will send logs/events ONLY IF they arrived to LM system via syslog protocol. The goal is to send all remaining events/logs to default vendor classifier template which will process and sort messages automatically. In order to achieve this, you must remove condition that limits processing only for syslog events.

Please edit your default syslog classifier by clicking on blue pencil and:

We recommend that you create backup of current configuration before changes. If something breaks you can always restore this version. You can do backup simply by copying XML code from XML tab and save it somewhere in your computer, for example like text file. If you need to recover it, simply copy all code from text file and paste it again in XML tab in classifier. To be more accurate - edit your default classifier, switch to XML tab, select all code here, delete and paste there your backuped code.

1. Rename it to vendor-Default

2. Remove the “if part” - drag the block with “if do” and disconnected it from “Process as:” Also drag the “pass to template” block and disconnected it from “if do” block. Now it looks like this:

   

3. Now connect block “pass to template” to “Process as:”

4. Delete whole "if do" block.

   

5. Save it by clicking on Save button.

6. Now all your logs which are not only from syslog, will be classified in vendor-Default classification template which will be useful in the future.

You can also edit classifier by clicking on XML a insert this code:

<xml xmlns="http://www.w3.org/1999/xhtml">
<variables></variables>
<block type="def" id="1" deletable="false" x="60" y="20">
<comment pinned="false" h="80" w="160">Describe this function...</comment>
<statement name="STACK">
  <block type="classifier_pass_to_template" id="bRq6VsZ=k4vt(_f{qWpr">
    <field name="TARGET">fc2518e1-fec5-4106-b176-2e10c1866df0</field>
  </block>
</statement>
</block>
</xml>

Don’t forget to rename the classifier, XML code does not include Name, type or description.

There is also second option – add new classification, but it is not recommended for the future purposes and it also has impact on system resources:

1. add new Classifier rule named “beats” with this condition below.

   

Also ask Your Logmanager certified partner to review, update and clean-up Your classification rules prior starting to use Beats Agents. Since release 3.5.0 we changed how data is processed and newer aproach is much more simple and easy to understand.

If You do not have Logmanager certified partner, open a support ticket with vendor via email: help@logmanager.com, we will promptly assist.

Additional explanation: Classifiers are under strict control of the customer/partner to match specific environment needs. Each Logmanager installation is unique. Therefore, vendor cannot update any Classifier automatically without the risk of breaking intended functionality configured by customer. For vendor created classification, we use Classifier Templates that are locked for user editation and therefore can be updated by vendor. Logmanager with initial software version 3.4.x prior any software upgrade is missing last resort rule for all the traffic (including Beats Agents traffic introduced in version 3.8.x and above). As the classification can impact overall performance of the Logmanager appliance, it makes sense to periodically review the classifier rules and optimize/cleanup them for current customer needs.