Cluster

Cluster mode currently supports running on up to 8 servers in Master - Slave mode. One server is designated as the primary (Master), the other servers are designated as secondary (slave).

All servers contain identical data.

The master server is the server from which the subordinate server has been created.

All database data and configuration are always replicated from Master to Slave.

It takes up to 5 minutes to synchronize a configuration after it has been saved. For example, if you change a classification rule on the master node, expect the configuration to be written to the slave nodes within 5 minutes.

If connectivity is lost between nodes for any amount of time:

Slave automaticaly reconnects to master when connectivity is restored and sync all changes from master to slave.
When slave node cannot reach master node, user is unable to do any search operations on slave, and slave node is not storing any data to database.
Incoming events in this scenario are automatically parsed and queued on disk. Queued data is automatically stored to the database when the master node is available.
Logmanager slave automatically sends periodic emails to the system admin about cluster disconnect (every 2 minutes).

Do not keep cluster disconnected for long period of time! Any cluster issue should be resolved within a matter of days at worst.

The main table displays information about Hostname, IP address and Status of the each node. If the Cluster mode is activated, then status and information about the joined (master or slave) server is displayed.

To know how to set your network connection correctly, you can see Logmanager with High Availability

Creating new cluster

Before you create the cluster, you have to consider which server you want to use as the main (master).

Cluster network connectivity requirements:

Cluster members can be placed in different IP subnets/L3 networks (buildings, city etc.).
The network connectivity available for cluster synchronization must be 1Gbit/s for a 2-node cluster and 10Gbit/s for a multi-node cluster.
The network latency between nodes must not exceed 10ms.

We recommend that all the logs (sources) used in correlation use cases deployed on cluster, are pointed to the same Logmanager cluster member.

Both servers must be running. Each server must have its own IP address and set the allowed ports according to the schema: Communication of Logmanager.

On the server that you selected as the master, create the Cluster by clicking on the plus icon and enter the following information into the form Connect Node:

Hostname: Enter the name of the subordinate server.
IP address: Enter the IP address of the subordinate server.
User Name: Enter the administrator name of the subordinate server.
Password: Enter the administrator password of the subordinate server.

After a successful activation, the status of the connected server will be displayed in the overview.

Slave node will automatically reboot and connect to the master.

After a succesfull connection, the slave node will automatically start to download all configuration/database data from the master. Please note that in the case of longer running LM server (the database contains multiple TB of data), initial synchronization can take a couple of weeks.

Master node will overwrite all configuration and stored data on slave node.

Editing existing cluster

If it’s necessary to change the settings for the cluster, on the master server, you can perform editing parameters of connection to the slave server. You can only set the IP address. If you need to change more parameters, it is necessary to disconnect the slave server, and then again you must join the servers.

To edit the existing cluster, click on the blue pencil icon.

Disconnect existing cluster

For disconnect the cluster click on the cross icon.

When disconnecting cluster from slave node, you must manually restart slave node for changes to take effect.

Recovery workflow on cluster HW failure / disaster recovery

In case of master node hardware failure (in case when server is completely dead/destroyed):

Log in to the remaining cluster slave node.
Disconnect the cluster master node (menu System/Cluster).
Manually reboot the slave node. After rebooting, the slave node boots as new individually running Logmanager.
Wait for the master node HW to be replaced.
Create a new cluster. Please note that the old slave box has all the data and must be the master node in the new cluster so everything can be replicated to the replacement node.

In case of slave node hardware failure (the server is completely dead/destroyed):

Log in to the master node.
Delete the old nonworking slave node (System/Cluster).
Wait for the slave node HW to be replaced (there is no configuration needed in advance).
Create a new cluster.