Skip to main content
Version: 3.25

Test Disaster Recover (DR)

Requirements
  • Have two senhasegura instances.
  • Both instances in the cluster and working correctly - check Enable cluster (HA/DR).
  • Snapshot from both instances.

This test is for two cluster members. You can check if at the bottom of each member if they have the following tags:

Primary member

Application

  • Production
  • Enabled

Replication

  • Master

Secondary member (Contingency)

Application

  • Contingency
  • Disabled

Replication

  • Slave

Preparation​

For the test to work, it will be necessary to make an abrupt shutdown in the primary member.

caution

Remember always to create a snapshot before this test. Since we are doing an abrupt shutdown, this can cause damage to the instance.

Cluster Validation​

Go to the Orbit ➔ Replication ➔ Status.

In the Database cluster table, check if the Cluster size matches the number of members in the cluster, and in the Cluster members table, remember that this test will be 2.

Configure Recovery settings​

In the secondary member, go to cluster Orbit ➔ Settings ➔ Recovery.

  • Insert the IPs allowed to perform a system recovery. You can use an IP or a subnet mask, e.g., 192.168.1.2, 192.168.1.0/24. Wildcards are not allowed, use * will not work.
    info

    This list will allow the users to see the button "Assume as master" turning the secondary member as Master and functional instance.

Execute DR Test​

  1. Force an abrupt shutdown in your primary member.

    caution

    Be aware that it must be an abrupt shutdown; otherwise, the cluster will know that one of the members turned it off, and the secondary member will not show the Recovery page.

  2. Once the primary member is down due to unexpected behavior, the secondary member will enter in split-brain, blocking any database change until manual instructions, and show the Recovery page on the web application.

  3. At the web application, click the Assume as master button, click Yes to confirm, setting the secondary member as a new Master.

    This process can take a few minutes.

    info

    If the button does not show, check the Configure Recovery settings to see if there are the correct IPs.

  4. Once the Orbit Web interface is available in the secondary member, check the tag showing that this instance is now the Master.

  5. To access the other senhasegura modules, enabling the application is necessary. Go to Orbit ➔ Settings ➔ Application toggle the "Enable application" button, and click save at the button. It showing green means that the application is now enabled.

  6. Log off and log in again to be able now to access the other modules.

Now all the senhasegura features are available and functional in the DR secondary member.

Primary Member Recovery​

After the steps executed in the step before, perform the following actions:

  1. Turn the primary member on, and wait for the synchronization with the other member cluster database. This can take a few minutes.

    The primary member will identify that something went wrong, and now the second member is the master and automatically synchronize the new information between the members.

  2. Once done, the login page should be displayed in the primary web application interface.

  3. Log in to the primary member web application and click Assume as master to return the primary member as Master.

  4. In the secondary member, go to Orbit ➔ Settings ➔ Application toggle the "Enable application" button to disable the application, and click save at the button. Make sure that it is not showing green.

Alternative Primary Member Recovery via SSH​

  1. Log in using SSH at port 59022 to the primary member with the mt4adm user.

  2. Execute the command sudo orbit application status, to check the information Main and primary.

$ sudo orbit application status

Application: Active
Replication: Active
Instance:    Cluster
Primary:     member2
Main:        No

  1. Execute the command sudo orbit application master.
$ sudo orbit application master

Application: Active
Replication: Active
Instance:    Cluster
Primary:     member1
Main:        Yes

This command will set the primary member to Main and redefine to Primary.