Cluster

Cluster mode currently supports running on up to 8 servers in Master - Slave mode. One server is designated as the primary (Master), the other servers designated as secondary (slave).

All servers contain identical data.

The master server is the server from which the subordinate server has been created.

All database data and configuration is always replicated from Master to Slave.

It takes up to 5 minutes to synchronize a configuration after it has been saved. For example, if you change a classification rule on the master node, expect the configuration to be written to the slave nodes within 5 minutes.

If connectivity is lost between nodes for any amount of time:

• slave automaticaly reconnects to master when connectivity is restored and sync all changes from master to slave.
• when slave node cannot reach master node, user is unable to do any search operations on slave, and slave node is not storing any data to database.
• Incoming events in this scenario are automatically parsed and queued on disk. Queued data is automatically stored to database when master node is available.
• Logmanager slave automatically sends periodic emails to system admin about cluster disconnect (every 2 minutes) to notify admin about disconnected cluster.

Do not keep cluster disconnected for long period of times! Any cluster issue should be resolved in matter of days at worst case.

The main table displays information about Hostname, IP address and Status of the each node. If the Cluster mode is activated, then status and information about the joined (master or slave) server is displayed.

How to set your network connection correctly you can see Logmanager with High Availability

Before you create the cluster, you have to consider, which server you want to use as the main (master).

Cluster network connectivity requirements:

• Cluster members can be placed in different IP subnets/L3 networks (buildings, city etc.).
• The network connectivity available for cluster synchronization must be 1Gbit/s for a 2-node cluster and 10Gbit/s for a multi-node cluster.
• Network latency between nodes must not exceed 10ms.

We recommend that all the logs (sources) used in correlation use cases deployed on cluster are pointed to the same Logmanager cluster member.

Both servers must be running. Each server must have its own IP address and set the allowed ports according to the schema: Communication of Logmanager.

On the server that you selected as the master, create the Cluster by clicking on the plus icon and enter the following information into the form Connect Node:

• Hostname: Enter the name of the subordinate server.
• IP address: Enter the IP address of the subordinate server.
• User Name: Enter the administrator name of the subordinate server.
• Password: Enter the administrator password of the subordinate server.

Cluster mode is activated by clicking on the Connect button. Click on the Cancel button, to return to the overview Cluster without saving data.

After a successful activation, there will be displayed the status of the connected server in the list.

Slave node will automatically reboot and connect to master.

After succesfull connection, slave node will automatically start to download all configuration/database data from master. Please note, that in case of longer running LM server (database contains multiple TB of data) initial synchronization can take couple of weeks.

Master node will overwrite all configuration and stored data on slave node.

If it’s necessary to change the settings for the cluster, on the master server, you can perform editing parameters of connection to the slave server. You can only set the IP address. If you need to change more parameters, it is necessary to disconnect the slave server and then again you must join the servers.

To edit the existing cluster, click on the blue pencil icon. After the editing is complete, click on the Reconfiguration button for save the configuration or Cancel to return back.

For disconnect the cluster click on the cross icon. A window pops up where you confirm or cancel the disconnect.

When disconnecting cluster from slave node, you must manually restart slave node for changes to take effect.

In case of master node hardware failure (in case when server is completely dead/destroyed):

1. Login to remaining cluster slave node.
2. Disconnect cluster master node (menu System/Cluster).
3. Manually reboot slave node. After rebooting, the slave node boots as new individually running Logmanager.
4. Wait for master node HW to be replaced.
5. Create new cluster. Please note that old slave box has all the data and must be master node in new cluster, so everything can be replicated to replacement node.

In case of slave node hardware failure (in case when server is completely dead/destroyed):

1. Login to master node.
2. Delete old nonworking slave node (System/Cluster).
3. Wait for slave node HW to be replaced (there is no configuration needed to be in advance).
4. Create new cluster.