Chapter 3 Using MySQL Cluster Manager

Before you make any changes to your wild cluster, it is strongly recommended that you back it up using the ndb_mgm client. See Using The NDB Cluster Management Client to Create a Backup for more information.

Any cluster processes that are under the control of a system boot process management facility such as /etc/init.d on Linux systems or the Services Manager on Windows platforms should be removed from this facility's control. Consult your operating system's documentation for information about how to do this. Be sure not to stop any running cluster processes in the course of doing so.

Create a MySQL user named mcmd on each of the wild cluster's SQL nodes. It is required for MySQL Cluster Manager to run the import config and import cluster commands in the steps to follow. To create this user with root privileges on the SQL nodes of our sample wild cluster, log in to an SQL node with the mysql client as root and execute the SQL statements shown here:

CREATE USER 'mcmd'@'localhost' IDENTIFIED BY 'super';

GRANT ALL PRIVILEGES ON *.* TO 'mcmd'@'localhost' WITH GRANT OPTION;

Keep in mind that this must be done on all the SQL nodes, unless distributed privileges are enabled on the wild cluster.

Make sure every node of the wild cluster has been started with its node ID specified with the --ndb-nodeid option at the command line, not just in the cluster configuration file. That is required for each process to be correctly identified by mcmd during the import. You can check if the requirement is fulfilled by the ps -ef | grep command, which shows the options the process has been started with:

shell> ps -ef | grep ndb_mgmd
ari       8118     1  0 20:51 ?        00:00:04 /home/ari/bin/cluster/bin/ndb_mgmd --config-file=/home/ari/bin/cluster/wild-cluster/config.ini 
--configdir=/home/ari/bin/cluster/wild-cluster --initial --ndb-nodeid=50

(For clarity's sake, in the command output for the ps -ef | grep command in this and the upcoming sections, we are skipping the line of output for the grep process itself.)

If the requirement is not fulfilled, restart the process with the --ndb-nodeid option; the restart can also be performed in step (e) or (f) below for any nodes you are restarting in those steps.

Make sure that the configuration cache is disabled for each management node. Since the configuration cache is enabled by default, unless the management node has been started with the --config-cache=false option, you will need to stop and restart it with that option, in addition to other options that it has been started with previously.

On Linux, we can once again use ps to obtain the information we need to accomplish this step. In a shell on host 192.168.56.102, on which the management node resides:

shell> ps -ef | grep ndb_mgmd
ari       8118     1  0 20:51 ?        00:00:04 /home/ari/bin/cluster/bin/ndb_mgmd --config-file=/home/ari/bin/cluster/wild-cluster/config.ini 
--configdir=/home/ari/bin/cluster/wild-cluster --initial --ndb-nodeid=50

The process ID is 8118. The configuration cache is turned on by default, and a configuration directory has been specified using the --configdir option. First, terminate the management node using kill as shown here, with the process ID obtained from ps previously:

shell> kill -15 8118

Verify that the management node process was stopped—it should no longer appear in the output of another ps command.

Now, restart the management node as described previously, with the configuration cache disabled and with the options that it was started with previously. Also, as already stated in step (d) above, make sure that the --ndb-nodeid option is specified at the restart:

shell> /home/ari/bin/cluster/bin/ndb_mgmd --config-file=/home/ari/bin/cluster/wild-cluster/config.ini --config-cache=false  --ndb-nodeid=50
MySQL Cluster Management Server mysql-5.7.16-ndb-7.5.4
2016-11-08 21:29:43 [MgmtSrvr] INFO     -- Skipping check of config directory since config cache is disabled.

Verify that the process is running as expected, using ps:

shell> ps -ef | grep ndb_mgmd
ari      10221     1  0 19:38 ?        00:00:09 /home/ari/bin/cluster/bin/ndb_mgmd --config-file=/home/ari/bin/cluster/wild-cluster/config.ini --config-cache=false --ndb-nodeid=50

The management node is now ready for migration.

Kill each data node angel process using the system's facility. The angel process monitors a data node process during a cluster's operation and, if necessary, attempts to restart the data node process (see this FAQ for details) . Before a cluster can be imported, the angel processes must be stopped first. On a Linux system, you can identify angel processes by the output of the ps -ef command executed on the processes' host; here is an example of doing that on the host 192.168.56.103 of the sample cluster :

shell> ps -ef | grep ndbd
ari      12836     1  0 20:52 ?        00:00:00 ./bin/ndbd --initial --ndb-nodeid=2 --ndb-connectstring=192.168.56.102
ari      12838 12836  2 20:52 ?        00:00:00 ./bin/ndbd --initial --ndb-nodeid=2 --ndb-connectstring=192.168.56.102

While both the actual data node process and its angel process appear as processes ndbd, you can identify each by looking at the process IDs. The process ID of the angel process (italicized in the sample output above) appears twice in the command output, once for itself (in the first line of the output), and once as the ID of the parent process of the actual data node daemon (in the second line). Use the kill command to terminate the process with the identified process ID, like this:

shell> kill -9 12836

Verify that the angel process has been killed and the other ndbd process (the non-angel data node daemon) is still running by issuing the ps -ef command again, as shown here:

shell> ps -ef | grep ndbd
ari      12838     1  0 20:52 ?        00:00:02 ./bin/ndbd --initial --ndb-nodeid=2 --ndb-connectstring=192.168.56.102

Now repeat this process in a shell on host 192.168.56.104, as shown here:

shell> ps -ef | grep ndbd
ari      11274     1  0 20:57 ?        00:00:00 ./cluster//bin/ndbd --initial --ndb-nodeid=3 --ndb-connectstring=192.168.56.102
ari      11276 11274  0 20:57 ?        00:00:01 ./cluster//bin/ndbd --initial --ndb-nodeid=3 --ndb-connectstring=192.168.56.102

shell> kill -9 11274

shell> ps -ef | grep ndbd
ari      11276     1  0 20:57 ?        00:00:01 ./cluster//bin/ndbd --initial --ndb-nodeid=3 --ndb-connectstring=192.168.56.102

The wild cluster's data nodes are now ready for migration.

To check the PID file for the management node process, log in to a system shell on host 192.168.56.102, change to the management node's data directory as specified by the Datadir parameter in the cluster's configuration file, then check to see whether the PID file is present. On Linux, you can use the command shown here:

shell> ls ndb_*.pid
ndb_50.pid

Check the content of the matching .pid file using a pager or text editor. We use more for this purpose here:

shell> more ndb_50.pid
10221

The number shown should match the ndb_mgmd process ID. We can check this on Linux using the ps command:

shell> ps -ef | grep ndb_mgmd
ari      10221     1  0 19:38 ?        00:00:09 /home/ari/bin/cluster/bin/ndb_mgmd --config-file=/home/ari/bin/cluster/wild-cluster/config.ini --config-cache=false --ndb-nodeid=50

The management node PID file satisfies the requirements listed at the beginning of this section.

Next, we check the PID files for the data nodes, on hosts 192.168.52.103 and 192.168.52.104. Log in to a system shell on 192.168.52.103, then obtain the process ID of the ndbd process on this host, as shown here:

shell> ps -ef | grep ndbd
ari      12838     1  0 Nov08 ?        00:10:12 ./bin/ndbd --initial --ndb-nodeid=2 --ndb-connectstring=192.168.56.102

As specified in the cluster's configuration file, the node's DataDir is /home/ari/bin/cluster/wild-cluster/2/data. Go to that directory to look for a file named ndb_2.pid:

shell> ls ndb_*.pid
ndb_2.pid

Now check the content of this file, and you are going to see the process ID for angel process for the data node (see earlier instructions on stopping the angel process for more explanations on the angel process):

shell> more ndb_2.pid
12836

Change the number in the PID file to the data node's own PID:

shell> sed -i 's/12836/12838/' ndb_2.pid
shell> more ndb_2.pid
12838

Similarly, we locate and adjust the content of the PID file for the remaining data node (node ID 3, whose data directory is /home/ari/bin/cluster/wild-cluster/3/data) on host 192.168.52.104:

shell> ps -ef | grep ndbd
ari      11276     1  0 Nov09 ?        00:09:44 ./cluster//bin/ndbd --initial --ndb-nodeid=3 --ndb-connectstring=192.168.56.102

shell> more /home/ari/bin/cluster/wild-cluster/3/data/ndb_3.pid
11274

Edit the .pid file, so it contains the data node process's own PID:

shell> cd /home/ari/bin/cluster/wild-cluster/3/data/
shell> sed -i 's/11274/11276/' ndb_3.pid
shell> more ndb_3.pid
11276

The PID file for this data node also meets our requirements now, so we are ready to proceed to the mysqld node running on host 192.168.52.102.

To check the PID file for the mysqld node: the default location for it is the data directory of the node, specified by the datadir option in either a configuration file or at the command line at the start of the mysqld process. Let's go to the data directory /home/ari/bin/cluster/wild-cluster/51/data on host 192.168.52.104 and look for the PID file.

shell> ls *.pid
localhost.pid

Notice that the MySQL Server could have been started with the --pid-file option, which puts a PID file at a specified location. In the following case, the same mysqld node has been started with the mysqld_safe script, and the ps command reveals the value for the --pid-file used:

shell>  ps -ef | grep mysqld
ari      11999  5667  0 13:15 pts/1    00:00:00 /bin/sh ./bin/mysqld_safe --defaults-file=/home/ari/bin/cluster/wild-cluster.cnf --ndb-nodeid=51
ari      12136 11999  1 13:15 pts/1    00:00:00 /home/ari/bin/cluster/bin/mysqld --defaults-file=/home/ari/bin/cluster/wild-cluster.cnf 
--basedir=/home/ari/bin/cluster/ --datadir=/home/ari/bin/cluster/wild-cluster/51/data/ --plugin-dir=/home/ari/bin/cluster//lib/plugin 
--ndb-nodeid=51 --log-error=/home/ari/bin/cluster/wild-cluster/51/data//localhost.localdomain.err 
--pid-file=/home/ari/bin/cluster/wild-cluster/51/data//localhost.localdomain.pid

As in the example, it is likely that you have a PID file that is not named in the required format for cluster import (ndb_node_id.pid); and if the --pid-file option was used, the PID file might not be at the required location (the data directory). Let us look into the PID file being referred to in the last example:

shell> more /home/ari/bin/cluster/wild-cluster/51/data//localhost.localdomain.pid
12136

The PID file for the SQL node is at an acceptable location (inside the data directory) and has the correct contents (the right PID), but has the wrong name. Let us just copy the PID file into a correctly named file in the same directory, like this

shell> cd /home/ari/bin/cluster/wild-cluster/51/data/
shell> cp localhost.localdomain.pid ndb_51.pid

Install MySQL Cluster Manager and start mcmd on all hosts with the same system user who started the wild cluster processes. Once you have done this, you can start the mcm client (see Section 3.3, “Starting the MySQL Cluster Manager Client”) on any one of these hosts to perform the next few steps.

Create a MySQL Cluster Manager site encompassing all four of the wild cluster's hosts, using the create site command, as shown here:

mcm> create site --hosts=192.168.56.102,192.168.56.103,192.168.56.104 newsite;
+---------------------------+
| Command result            |
+---------------------------+
| Site created successfully |
+---------------------------+
1 row in set (0.15 sec)

We have named this site newsite. You should be able to see it listed in the output of the list sites command, similar to what is shown here:

mcm> list sites;
+---------+------+-------+----------------------------------------------+
| Site    | Port | Local | Hosts                                        |
+---------+------+-------+----------------------------------------------+
| newsite | 1862 | Local | 192.168.56.102,192.168.56.103,192.168.56.104 |
+---------+------+-------+----------------------------------------------+
1 row in set (0.01 sec)

Add a MySQL Cluster Manager package referencing the MySQL Cluster binaries using the add package command; use the command's --basedir option to point to the correct location of the MySQL Cluster executables. The command shown here creates such a package, named newpackage:

mcm> add package --basedir=/home/ari/bin/cluster newpackage;
+----------------------------+
| Command result             |
+----------------------------+
| Package added successfully |
+----------------------------+
1 row in set (0.70 sec)

You do not need to include the bin directory containing the MySQL Cluster executables in the --basedir path. If the executables are in /home/ari/bin/cluster/bin, it is sufficient to specify /home/ari/bin/cluster; MySQL Cluster Manager automatically checks for the binaries in a bin directory within the directory specified by --basedir.

Create the target cluster including at least some of the same processes and hosts used by the standalone cluster. Do not include any processes or hosts that are not part of this cluster. In order to prevent potentially disruptive process or cluster operations from interfering by accident with the import process, it is strongly recommended that you create the cluster for import using the --import option for the create cluster command.

You must also take care to preserve the correct node ID (as listed in the config.ini file shown previously) for each node.

The following command creates the cluster newcluster for import and includes the management and data nodes, but not the SQL or “free” API node (which we add in the next step):

mcm> create cluster --import --package=newpackage \
      --processhosts=ndb_mgmd:50@192.168.56.102,ndbd:2@192.168.56.103,ndbd:3@192.168.56.104 \
       newcluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.96 sec)

You can verify that the cluster was created correctly by checking the output of show status with the --process (-r) option, like this:

mcm> show status -r newcluster;
+--------+----------+----------------+--------+-----------+------------+
| NodeId | Process  | Host           | Status | Nodegroup | Package    |
+--------+----------+----------------+--------+-----------+------------+
| 50     | ndb_mgmd | 192.168.56.102 | import |           | newpackage |
| 2      | ndbd     | 192.168.56.103 | import | n/a       | newpackage |
| 3      | ndbd     | 192.168.56.104 | import | n/a       | newpackage |
+--------+----------+----------------+--------+-----------+------------+
3 rows in set (0.05 sec)

If necessary, add any remaining processes and hosts from the wild cluster not included in the previous step using one or more add process commands. We have not yet accounted for two of the nodes from the wild cluster: the SQL node with node ID 51, on host 192.168.56.102, and the API node with node ID 52, which is not bound to any specific host. You can use the following command to add both of these processes to newcluster:

mcm> add process --processhosts=mysqld:51@192.168.56.102,ndbapi:52@* newcluster;
+----------------------------+
| Command result             |
+----------------------------+
| Process added successfully |
+----------------------------+
1 row in set (0.41 sec)

Once again checking the output from show status -r, we see that the mysqld and ndbapi processes were added as expected:

mcm> show status -r newcluster;
+--------+----------+----------------+--------+-----------+------------+
| NodeId | Process  | Host           | Status | Nodegroup | Package    |
+--------+----------+----------------+--------+-----------+------------+
| 50     | ndb_mgmd | 192.168.56.102 | import |           | newpackage |
| 2      | ndbd     | 192.168.56.103 | import | n/a       | newpackage |
| 3      | ndbd     | 192.168.56.104 | import | n/a       | newpackage |
| 51     | mysqld   | 192.168.56.102 | import |           | newpackage |
| 52     | ndbapi   | *              | import |           |            |
+--------+----------+----------------+--------+-----------+------------+
5 rows in set (0.06 sec)

You can also see that since newcluster was created using the create cluster command's --import option, the status of all processes in this cluster—including those we just added—is import. This means we cannot yet start newcluster or any of its processes. The import status and its effects on newcluster and its cluster processes persist until we have completed importing another cluster into newcluster.

The target newcluster cluster now has the same processes, with the same node IDs, and on the same hosts as the original standalone cluster. We are ready to proceed to the next step.

Now, test first the effects of the import config command by running it with the --dryrun option (the step only works if you have created the mcmd user on the cluster's mysqld nodes):

mcm> import config --dryrun newcluster;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command result                                                                                                                                                                      |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Import checks passed. Please check /home/ari/bin/mcm_data/clusters/newcluster/tmp/import_config.49d541a9_294_0.mcm on host localhost.localdomain for settings that will be applied. |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (6.87 sec)

As indicated by the output from import config --dryrun, you can see the configuration attributes and values that would be copied to newcluster by the command without the --dryrun option in the file /path-to-mcm-data-repository/clusters/clustername/tmp/import_config.message_id.mcm. If you open this file in a text editor, you will see a series of set commands that would accomplish this task, similar to what is shown here:

# The following will be applied to the current cluster config:
set NoOfReplicas:ndbd=2 newcluster;
set DataDir:ndb_mgmd:50=/home/ari/bin/cluster/wild-cluster/50/data newcluster;
set DataDir:ndbd:2=/home/ari/bin/cluster/wild-cluster/2/data newcluster;
set DataDir:ndbd:3=/home/ari/bin/cluster/wild-cluster/3/data newcluster;
set basedir:mysqld:51=/home/ari/bin/cluster/ newcluster;
set datadir:mysqld:51=/home/ari/bin/cluster/wild-cluster/51/data/ newcluster;
set sql_mode:mysqld:51="NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES" newcluster;
set ndb_connectstring:mysqld:51=192.168.56.102 newcluster;

After the successful dry run, you are now ready to import the wild cluster's configuration into newcluster, with the command shown here:

mcm> import config newcluster;
+------------------------------------------------------------------------------------------------------------------+
| Command result                                                                                                   |
+------------------------------------------------------------------------------------------------------------------+
| Configuration imported successfully. Please manually verify plugin options, abstraction level and default values |
+------------------------------------------------------------------------------------------------------------------+

As an alternative, instead of importing all the settings using the import config command, you can make changes to the /path-to-mcm-data-repository/clusters/clustername/tmp/import_config.message_id.mcm file generated by the dry run as you wish, and then import the settings by executing the file with the mcm agent:

mcm> source /path-to-mcm-data-repository/clusters/clustername/tmp/import_config.message_id.mcm

You should check the resulting configuration of newcluster carefully against the configuration of the wild cluster. If you find any inconsistencies, you must correct these in newcluster using the appropriate set commands.

Perform a test run of the proposed import using import cluster with the --dryrun option. When this option is used, MySQL Cluster Manager checks for mismatched configuration attributes, missing or invalid processes or hosts, missing or invalid PID files, and other errors, and warns of any it finds, without actually performing any migration of processes or data (the step only works if you have created the mcmd user on the cluster's mysqld nodes):

mcm> import cluster --dryrun newcluster;

If errors occur, correct them, and repeat the dry run shown in the previous step until it returns no more errors. The following list contains some common errors you may encounter, and their likely causes:

When import cluster --dryrun no longer warns of any errors, you can perform the import with the import cluster command, this time omitting the --dryrun option.

mcm> import cluster newcluster;
+-------------------------------+
| Command result                |
+-------------------------------+
| Cluster imported successfully |
+-------------------------------+
1 row in set (5.58 sec)

You can check that the wild cluster has now been imported, and is now under management of MySQL Cluster Manager:

mcm> show status -r newcluster;
+--------+----------+----------------+---------+-----------+------------+
| NodeId | Process  | Host           | Status  | Nodegroup | Package    |
+--------+----------+----------------+---------+-----------+------------+
| 50     | ndb_mgmd | 192.168.56.102 | running |           | newpackage |
| 2      | ndbd     | 192.168.56.103 | running | 0         | newpackage |
| 3      | ndbd     | 192.168.56.104 | running | 0         | newpackage |
| 51     | mysqld   | 192.168.56.102 | running |           | newpackage |
| 52     | ndbapi   | *              | added   |           |            |
+--------+----------+----------------+---------+-----------+------------+
5 rows in set (0.01 sec)

Identify the backup to be used.

In this example, we use the backup having the ID 4, that was created for mycluster previously in this section.

Wipe the MySQL Cluster data.

The simplest way to do this is to stop and then perform an initial start of the cluster as shown here, using mycluster:

mcm> stop cluster mycluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster stopped successfully |
+------------------------------+
1 row in set (15.24 sec)

mcm> start cluster --initial mycluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (34.47 sec)

Restore the backup.

This is done using the restore cluster command, which requires the backup ID and the name of the cluster as arguments. Thus, you can restore backup 4 to mycluster as shown here:

mcm> restore cluster --backupid=4 mycluster;
+--------------------------------+
| Command result                 |
+--------------------------------+
| Restore completed successfully |
+--------------------------------+
1 row in set (16.78 sec)

Locate the logical backup for the metadata; see Locations of backup files in Section 3.6.2.5, “Restoring a Backup to a Cluster with Fewer Data Nodes” for instructions.

Restore the logical backup; see Restore the Logical Backup of NDB Table Metadata in Section 3.6.2.5, “Restoring a Backup to a Cluster with Fewer Data Nodes” for instructions.

Stop the original cluster:

mcm> stop cluster mycluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster stopped successfully |
+------------------------------+
1 row in set (19.54 sec)

mcm> show status mycluster;
+-----------+---------+---------+
| Cluster   | Status  | Comment |
+-----------+---------+---------+
| mycluster | stopped |         |
+-----------+---------+---------+
1 row in set (0.05 sec)

Start your new cluster. Make sure the new cluster is operational and it has at least one free ndbapi slot for the ndb_restore utility to connect to the cluster:

mcm> start cluster newcluster2nodes;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (33.68 sec)

mcm> show status -r newcluster2nodes;
+--------+----------+---------+---------+-----------+-----------+
| NodeId | Process  | Host    | Status  | Nodegroup | Package   |
+--------+----------+---------+---------+-----------+-----------+
| 49     | ndb_mgmd | tonfisk | running |           | mypackage |
| 1      | ndbmtd   | tonfisk | running | 0         | mypackage |
| 2      | ndbmtd   | tonfisk | running | 0         | mypackage |
| 50     | mysqld   | tonfisk | running |           | mypackage |
| 51     | ndbapi   | *       | added   |           |           |
+--------+----------+---------+---------+-----------+-----------+
5 rows in set (0.09 sec)

Restore the logical backup of the metadata of the NDB tables onto the new cluster. See Reloading SQL-Format Backups for different ways to restore a logical backup. One way to do it is to open a mysql client, connect it to a mysqld node of the cluster, and then source the logical backup file with the mysql client:

mysql> source path-to-logical-backup-file/BACKUP-BackupID.mysqld_nodeid.schema.sql

See Locations of backup files above on how to find the path of the logical backup file. For our sample clusters, this is how the command looks like for restoring the NDB table metadata from the backup with the BackupID 2:

mysql> source /opt/mcmbackup/BACKUP/BACKUP-2/BACKUP-2.50.schema.sql

Restore one by one the backup for each data node of the original cluster to the new cluster, using the ndb_restore program:

shell> ndb_restore -b BackupID -n nodeID -r --backup_path=backup-folder-for-data_node

See Locations of backup files above on how to find the path of the data node backup files. For our sample clusters, to restore the data from the backup with the BackupID 2 for data node 1 to 4 of mycluster, execute the following commands:

shell> ndb_restore --backupid=2 --nodeid=1 --restore_data --backup_path=/opt/mcmbackup/BACKUP/BACKUP-2/ --disable-indexes

shell> ndb_restore --backupid=2 --nodeid=2 --restore_data --backup_path=/opt/mcmbackup/BACKUP/BACKUP-2/ --disable-indexes

shell> ndb_restore --backupid=2 --nodeid=3 --restore_data --backup_path=/opt/mcmbackup/BACKUP/BACKUP-2/ --disable-indexes

shell> ndb_restore --backupid=2 --nodeid=4 --restore_data --backup_path=/opt/mcmbackup/BACKUP/BACKUP-2/ --disable-indexes

The option --disable-indexes is used so indexes are ignored during the restores. This is because if we also try to restore the indexes node by node, they might not be restored in the right order for the foreign keys and unique key constraints to work properly. Therefore, the --disable-indexes option is used in the above commands, after the execution of which we rebuild the indexes with the following ndb_restore command and the --rebuild-indexes option (you only need to run this on one of the data nodes):

shell> ndb_restore --backupid=2 --nodeid=1 --rebuild-indexes --backup_path=/opt/mcmbackup/BACKUP/BACKUP-2/