Logmanager log flow

This part of the documentation simplifies the description of log flow, which refers to the process of how logs/events are handled on the Logmanager side. It explains where meta data, for example, is generated or where message data is located.

---
title: Logmanager
---

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#fff',
      'primaryTextColor': '#0F0FD7',
      'primaryBorderColor': '#0F0FD7',
      'lineColor': '#0F0FD7',
      'secondaryColor': '#fff',
      'tertiaryColor': '#fff'
    },
    'flowchart' : { 

    }
  }
}%%

flowchart 

  A[|borders:no|"Sending the log to Logmanager"]   ~~~ 
  B[|borders:no|"Log is saved to Buffer in raw,
@timestamp and _id are created"]  ~~~ 
  C[|borders:no|" "] ~~~
  E[|borders:no|"Meta data are created"] ~~~ 
  D[|borders:no|"Msg data are created"] ~~~ 
  F[|borders:no|"Correlation, email sending"] ~~~ 
  G[|borders:no|"Save to database"] 

  A1([Log])-->B1(Buffer)
  B1(Buffer)-->C1(Classification)
  C1(Classification)-->D1(Parsing)
  C1(Classification)--No-->E1(Classifier Templates)
  E1(Classifier Templates)--Yes-->D1(Parsing)
  E1(Classifier Templates)--No-->F1(Alert, correlation)
  D1(Parsing)-->F1(Alert, correlation)
  F1(Alert, correlation)-->G1[(Database)]
  F1(Alert, correlation)-->H1(Syslog Output)

• Log: Receiveing logs to Logmanager.
• Buffer: The log is stored in a buffer, where it is assigned a timestamp (@timestamp) and a unique ID (_id). The log is also stored in the raw format. If the buffer is nearly at its capacity, an email is sent to the operator, indicating potential overload of Logmanager.
• Classification: In classification, meta data is generated, such as meta.src.ip, which identifies the IP address of the device that generated the log, or meta.parser, which indicates the parser that will process the log in the next step. Meta data is generated automatically and determines the origin of the log, the device that sent it, but does not contain information about the event itself. Through custom classification configuration, you can determine how logs are handled. If no classification is assigned to a log, it is passed to classification templates. For more information about classification, refer to Classifiers.
• Classification templates: If the log does not pass classification, it is processed by classification templates defined by the manufacturer. This step involves the use of preconfigured IP Prefix lists - IP Prefix lists for additional source identification. If the source is still not identified in the classification templates and does not have a parser assigned, it proceeds to alerts, correlations. Otherwise, it is processed by a parser. For more information about classification templates, refer to Classifier Templates.
• Parsing: In this step, logs are processed by a parser, where message data are created. Message data contains information about the event within the log itself. Message data are generated only if the log is directed to one of the available parsers as part of the classification. Message data may include, for example, msg.device_id, and so on. The creation of message data depends on the specific parser used, so it may vary in practice. As a general rule, if logs in Logmanager do not have message data, it is likely that they were not sent to the parser (there may be an issue with incorrect classification). You can verify this by checking the log, where you will see meta.parser=unknown. For more information about parsing, refer to Parsers.
• Alerts, correlations: In this part, correlation rules and alerts are triggered. Alerts can also be used to enrich logs by creating new message data without editing the parser. When an email is sent for an alert, it internally assignes the system tag notified which is used to display all sent alerts in the Alerted Events dashboard. For more information about alerts, refer to Alerts.
• Syslog output: In this step, the log is simultaneously forwarded to another syslog server if you have one configured using the syslog forwarding feature. The format of the forwarded log is determined by the operator in the settings - Syslog Forwarding.
• Database: The log is finally stored in the database and ready for further analysis and visualization. In the database, each log, which includes meta data, message data, and raw data, is stored in a JSON format.

• Regardless of classification/parsing, each log is stored in the database in its raw form.
• The database is designed to prevent modification or deletion of stored logs.
• The @timestamp is generated based on the ingress time on Logmanager. To ensure data credibility, Logmanager does not rely on any timestamp mentioned in the log header.