Parsers

This section is used to define source devices from which logs are retrieved. The device can be hardware but also software, including Windows. Every device you set to send logs to Logmanager, need to be defined in this section.

Definition is divided into several groups. First, classification of incoming messages is dealt with. Classification is set, according to identifiers that each message contains. Message is transferred from the classification to the parser, where its content is processed.

Remaining items in this section include enhanced features for classification and parsers.

Every message, which goes thru the system, is saved into the database in standardized format. It is possible to work with messages in system based on this format, therefore is appropriate to know storage and meaning of specific components of the message.

Messages use following syntax:

{
  "field": "value"
}

{
  "field1": "string_value",
  "field2": integer_value,
  "field3": float_value,
  "field4": [
    list
  ],
  "field5": {
    dictionary
  },
  "field6": {
    "subfield1": "value1",
    "subfield2": "value2"
  }
}

Examples:

• string value: "sample text",
• integer value: 259,
• float value: 259.57,
• list: ["sample text", 259, ""],
• dictionary: {"field1": "sample text", "field2": 259},
• get value from subfield2: field6.subfield2.

Examples of correct field names:

• protocol
• url_path
• service-internal-1
• service_internal_1

Examples of incorrect field names:

Following rules are applied on messages:

• Field name is always only in lowercase letters, completed by numbers, dash and the underline.
• If the field is named incorrectly, it will be automatically cleared from wrong characters and completed by a text invalid_.
• Value can be empty, integer, float, string, list or a dictionary.
• Float values cannot be NaN or infinite.
• Fields which have @ in name:
  • List’s are internally generated.
  • @ or _ symbol cannot be used as variable name in parsers or classifiers.
  • List’s may contain additional information in list or dictionary format.

General format of saved message:

{
  "@timestamp": "",
  "raw": "",
  "meta": {
    "event@id": "",
    "forwarder@id": "",
    "instance@id": "",
    "parser": "",
    "plugin": "",
    "tags": [],
    "tags@id": [],
    "timestamp": "",
    "src": {},
    "type": ""
  },
  "msg": {},
  "raw_offset": 0,
  "@version": ""
}

Format description:

• @timestamp: timestamp, when the message was delivered to the Logmanager system,
• raw: original contents of the source message,
• meta: contains internal information about the message:
  • event@id: automaticaly generated unique ID for each event,
  • forwarder@id: identification of Forwarder, which sent the message to Logmanager server,
  • instance@id: identification of instance, to which the message was delivered,
  • parser: name of the parser, which processed the message,
  • plugin: name of the source, from which the message was delivered,
  • tags: list of tag values, which are used on the message,
  • tags@id: list of tag identifier, which are used on the message,
  • timestamp: timestamp, when the message was delivered to the Logmanager system,
  • src: contains source information from where the message was delivered to Logmanager system (IP address, protocol),
• type: indicates internal message status, if the message went thru a parser,
• msg: contains a dictionary of fields generated based on parsers, which the message passed thru,
• raw_offset: determines position in text, where from is the message processed by parsers,
• @version: internal version of message in the system.

When working with the messages use only values listed in fields msg, meta, raw and raw_offset. DOther fields are system fields and can be changed in next Logmanager versions. So, it’s not recommended to work with them.

{
  "_index": "lm-demo-2017.01.26",
  "_type": "microsoft-windows",
  "_id": "AVnZ_79FTDlIxPtA3bf8",
  "_score": null,
  "_source": {
    "msg": {
      "eventid": "7036",
      "processid": "500",
      "systemtime": "2017-01-26T08:58:00.078932400Z",
      "guid": "{555908d1-a6d7-4695-8e1e-26931d2012f4}",
      "uuid": "34B64D56-E785-1328-0EC0-E6CD20C4A9F9",
      "osservicepack": "",
      "eventsourcename": "Service Control Manager",
      "eventrecordid": "140740",
      "osname": "Windows 8 or Windows Server 2012",
      "eventtype": "System",
      "osedition": "Standard Server",
      "computer": "pc-example",
      "param2": "stopped",
      "param1": "Adobe Flash Player Update Service",
      "threadid": "2384",
      "message": "The Adobe Flash Player Update Service service entered the stopped state.",
      "eventid@int": {
        "value": 7036
      },
      "senderversion": "2.4.6004.23920",
      "channel": "System",
      "name": "Service Control Manager"
    },
    "raw": "<36>1 2017-01-26T09:58:01.094556+01:00 pc-example ServiceControlManager 500 msgld {\"senderversion\": \"2.4.6004.23920\", \"eventtype\": \"System\", \"eventid\": \"7036\", \"eventrecordid\": \"140740\", \"osname\": \"Windows 8 or Windows Server 2012\", \"osedition\": \"Standard Server\", \"osservicepack\": \"\", \"uuid\": \"34B64D56-E785-1328-0EC0-E6CD20C4A9F9\", \"name\": \"Service Control Manager\", \"guid\": \"{555908d1-a6d7-4695-8e1e-26931d2012f4}\", \"eventsourcename\": \"Service Control Manager\", \"systemtime\": \"2017-01-26T08:58:00.078932400Z\", \"processid\": \"500\", \"threadid\": \"2384\", \"channel\": \"System\", \"computer\": \"pc-example\", \"param1\": \"Adobe Flash Player Update Service\", \"param2\": \"stopped\", \"message\": \"The Adobe Flash Player Update Service service entered the stopped state.\"}",
    "@timestamp": "2017-01-26T08:58:03.595+00:00",
    "meta": {
      "forwarder@id": "00000000-0000-0000-0000-000000000000",
      "tags": ["windows"],
      "timestamp": "2017-01-26T08:58:03.595836+00:00",
      "parser": "microsoft-windows",
      "tags@id": ["3d3f2c7f-d1db-4442-9ff2-ea60c56716a8"],
      "instance@id": "00000000-0000-0000-0000-000000000000",
      "src": {
        "dialect": "relp",
        "severity": "warning",
        "facility": "auth",
        "ip": "192.0.2.56",
        "ip@ip": {
          "city": "New York",
          "is_reserved": false,
          "value": "192.0.2.56",
          "version": 4,
          "country_code": "US",
          "is_multicast": false,
          "country_name": "United States",
          "ptr": "pc-example.example.com",
          "is_link_local": false
        },
        "pid": "500",
        "host": "pc-example",
        "program": "ServiceControlManager"
      },
      "type": "user",
      "plugin": "windows"
    },
    "raw_offset": 82,
    "@version": "1"
  }
}

Description of example message:

• processed information from the source message are stored in field msg,
• original source message is show in field raw,
• information from where the message was sent, is show in field src.