Chapter 3 Using MySQL Cluster Manager

Install MySQL Cluster Manager and start mcmd on all hosts; we assume that you have installed MySQL Cluster Manager to the recommended location, in this case the directory /opt/mcm-1.3.6. (See Section 2.3, “MySQL Cluster Manager Installation”, for more information.) Once you have done this, you can start the mcm client (see Section 3.3, “Starting the MySQL Cluster Manager Client”) on any one of these hosts to perform the next few steps.

Create a MySQL Cluster Manager site encompassing all four of these hosts, using the create site command, as shown here:

mcm> create site --hosts=alpha,beta,gamma,delta newsite;
+---------------------------+
| Command result            |
+---------------------------+
| Site created successfully |
+---------------------------+
1 row in set (0.15 sec)

We have named this site newsite. You should be able to see it listed in the output of the list sites command, similar to what is shown here:

mcm> list sites;
+---------+------+-------+------------------------+
| Site    | Port | Local | Hosts                  |
+---------+------+-------+------------------------+
| newsite | 1862 | Local | alpha,beta,gamma,delta |
+---------+------+-------+------------------------+
1 row in set (0.01 sec)

Add a MySQL Cluster Manager package referencing the MySQL Cluster binaries, using the add package command; this command's --basedir option can be used to point to the correct location. The command shown here creates such a package, named newpackage:

mcm> add package --basedir=/usr/local/mysql newpackage;
+----------------------------+
| Command result             |
+----------------------------+
| Package added successfully |
+----------------------------+
1 row in set (0.70 sec)

You do not need to include the bin directory containing the MySQL Cluster executables in the --basedir path. Since the executables are in /usr/local/mysql/bin, it is sufficient to specify /usr/local/mysql; MySQL Cluster Manager automatically checks for the binaries in a bin directory within the one specified by --basedir.

Create the target cluster including at least some of the same processes and hosts used by the standalone cluster. Do not include any processes or hosts that are not part of this cluster. In order to prevent potentially disruptive process or cluster operations from interfering by accident with the import process, it is strongly recommended that you create the cluster for import, using the --import option for the create cluster command.

You must also take care to preserve the correct node ID (as listed in the config.ini file shown previously) for each node. In MySQL Cluster Manager 1.3.1 and later, using the --import option allows you to specify node IDs under 49 for nodes other than data nodes, which is otherwise prevented when using create cluster (the restriction has been lifted since MySQL Cluster Manager 1.3.4).

The following command creates the cluster newcluster for import, and includes the management and data nodes, but not the SQL or “free” API node (which we add in the next step):

mcm> create cluster --import --package=newpackage \
        --processhosts=ndb_mgmd:50@alpha,ndbd:1@beta,ndbd:2@gamma \
        newcluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster created successfully |
+------------------------------+
1 row in set (0.96 sec)

You can verify that the cluster was created correctly by checking the output of show status with the --process (-r) option, like this:

mcm> show status -r newcluster;
+--------+----------+-------+--------+-----------+------------+
| NodeId | Process  | Host  | Status | Nodegroup | Package    |
+--------+----------+-------+--------+-----------+------------+
| 50     | ndb_mgmd | alpha | import |           | newpackage |
| 5      | ndbd     | beta  | import | n/a       | newpackage |
| 6      | ndbd     | gamma | import | n/a       | newpackage |
+--------+----------+-------+--------+-----------+------------+
3 rows in set (0.01 sec)

If necessary, add any remaining processes and hosts from the “wild” cluster not included in the previous step using one or more add process commands. We have not yet accounted for 2 of the nodes from the wild cluster: the SQL node with node ID 100, on host delta; and the API node which has node ID 101, and is not bound to any specific host. You can use the following command to add both of these processes to newcluster:

mcm> add process --processhosts=mysqld:100@delta,ndbapi:101@* newcluster;
+----------------------------+
| Command result             |
+----------------------------+
| Process added successfully |
+----------------------------+
1 row in set (0.41 sec)

Once again checking the output from show status -r, we see that the mysqld and ndbapi processes were added as expected:

mcm> show status -r newcluster;
+--------+----------+-------+--------+-----------+------------+
| NodeId | Process  | Host  | Status | Nodegroup | Package    |
+--------+----------+-------+--------+-----------+------------+
| 50     | ndb_mgmd | alpha | import |           | newpackage |
| 5      | ndbd     | beta  | import | n/a       | newpackage |
| 6      | ndbd     | gamma | import | n/a       | newpackage |
| 100    | mysqld   | delta | import |           | newpackage |
| 101    | ndbapi   | *     | import |           |            |
+--------+----------+-------+--------+-----------+------------+
5 rows in set (0.08 sec)

You can also see that, since newcluster was created using the create cluster command's --import option, the status of all processes in this cluster—including those we just added—is import. This means we cannot yet start newcluster or any of its processes, as shown here:

mcm> start process 50 newcluster;
ERROR 5317 (00MGR): Unable to perform operation on cluster created for import
mcm> start cluster newcluster;
ERROR 5317 (00MGR): Unable to perform operation on cluster created for import

The import status and its effects on newcluster and its cluster processes persist until we have completed importing another cluster into newcluster.

Following the execution of the add process command shown previously, the target newcluster cluster now has the same processes, with the same node IDs, and on the same hosts as the original standalone cluster. We are ready to proceed to the next step.

Now it is necessary to duplicate the wild cluster's configuration attributes in the target cluster. In MySQL Cluster Manager 1.3.1 and later, you can handle most of these using the import config command, as shown here:

mcm> import config --dryrun newcluster;
+---------------------------------------------------------------------------+
| Command result                                                            |
+---------------------------------------------------------------------------+
| Import checks passed. Please check log for settings that will be applied. |
+---------------------------------------------------------------------------+
1 row in set (5.36 sec)

As indicated by the output from import config --dryrun, you can see the configuration attributes and values that would be copied to newcluster by the unimpeded command in the agent log file (mcmd.log), which by default is created in the MySQL Cluster Manager installation directory. If you open this file in a text editor, you can locate a series of set commands that would accomplish this task, similar to what is shown here (emphasized text):

2014-03-14 16:05:11.896: (message) [T0x1ad12a0 CMGR ]: Got new message mgr_import_configvalues {84880f7a 35 0}
2014-03-14 16:05:11.896: (message) [T0x1ad12a0 CMGR ]: Got new message mgr_import_configvalues {84880f7a 36 0}
2014-03-14 16:05:11.896: (message) [T0x1ad12a0 CMGR ]: Got new message mgr_import_configvalues {84880f7a 37 0}
2014-03-14 16:05:13.698: (message) [T0x7f4fb80171a0 RECFG]: All utility process have finished
2014-03-14 16:05:13.698: (message) [T0x7f4fb80171a0 RECFG]: Process started : /usr/local/mysql/bin/mysqld --no-defaults --help --verbose
2014-03-14 16:05:13.698: (message) [T0x7f4fb80171a0 RECFG]: Spawning mysqld --nodefaults --help --verbose asynchronously
2014-03-14 16:05:13.904: (message) [T0x7f4fb80171a0 RECFG]: Successfully pulled default configuration from mysqld 100
2014-03-14 16:05:13.905: (warning) [T0x7f4fb80171a0 RECFG]: Failed to remove evsource!
2014-03-14 16:05:15.719: (message) [T0x7f4fb80171a0 RECFG]: All utility process have finished
2014-03-14 16:05:15.725: (message) [T0x7f4fb80171a0 RECFG]: Applying mysqld configuration to cluster...
2014-03-14 16:05:16.186: (message) [T0x1ad12a0 CMGR ]: Got new message mgr_import_configvalues {84880f7a 38 0}
2014-03-14 16:05:16.187: (message) [T0x1ad12a0 CMGR ]: Got new message x_trans {84880f7a 39 0}
2014-03-14 16:05:16.286: (message) [T0x1ad12a0 CMGR ]: Got new message x_trans {84880f7a 40 0}
2014-03-14 16:05:16.286: (message) [T0x7f4fb80171a0 RECFG]: The following will be applied to the current cluster config:
set DataDir:ndb_mgmd:50="" newcluster
set IndexMemory:ndbd:5=1073741824 newcluster
set DataMemory:ndbd:5=1073741824 newcluster
set DataDir:ndbd:5=/usr/local/mysql/mysql-cluster/data newcluster
set ThreadConfig:ndbd:5="" newcluster
set IndexMemory:ndbd:6=1073741824 newcluster
set DataMemory:ndbd:6=1073741824 newcluster
set DataDir:ndbd:6=/usr/local/mysql/mysql-cluster/data newcluster
set ThreadConfig:ndbd:6="" newcluster
set basedir:mysqld:100=/usr/local/mysql newcluster
set character_sets_dir:mysqld:100=/usr/local/mysql/share/charsets newcluster
set datadir:mysqld:100=/usr/local/mysql/data newcluster
set general_log_file:mysqld:100=/usr/local/mysql/data/delta.log newcluster
set lc_messages_dir:mysqld:100=/usr/local/mysql/share newcluster
set log_error:mysqld:100=/usr/local/mysql/data/delta.err newcluster
set ndb_connectstring:mysqld:100=alpha newcluster
set ndb_mgmd_host:mysqld:100=alpha newcluster
set optimizer_trace:mysqld:100=enabled=off,one_line=off newcluster
set pid_file:mysqld:100=/usr/local/mysql/data/delta.pid newcluster
set plugin_dir:mysqld:100=/usr/local/mysql/lib/plugin newcluster
set report_port:mysqld:100=3306 newcluster
set slow_query_log_file:mysqld:100=/usr/local/mysql/data/delta-slow.log newcluster
set sql_mode:mysqld:100=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION newcluster

Assuming that the dry run was successful, you should now be able to import the wild cluster's configuration into newcluster, with the command and a result similar similar to what is shown here:

mcm> import config newcluster;
+------------------------------------------------------------------------------------------------------------------+
| Command result                                                                                                   |
+------------------------------------------------------------------------------------------------------------------+
| Configuration imported successfully. Please manually verify plugin options, abstraction level and default values |
+------------------------------------------------------------------------------------------------------------------+

You should check the log from the dry run and the resulting configuration of newcluster carefully against the configuration of the wild cluster. If you find any inconsistencies, you must correct these in newcluster using the appropriate set commands afterwards.

MySQL Cluster Manager acts through a MySQL user named mcmd on each SQL node. It is therefore necessary to create this user and grant root privileges to it. To do this, log in to the SQL node running on host delta and execute in the mysql client the SQL statements shown here:

CREATE USER 'mcmd'@'delta' IDENTIFIED BY 'super';

GRANT ALL PRIVILEGES ON *.* TO 'mcmd'@'delta' WITH GRANT OPTION;

Keep in mind that, if the “wild” cluster has more than one SQL node, you must create the mcmd user on every one of these nodes.

Kill each data node angel process using the system's facility for doing so. Do not kill any non-angel data node daemons. On a Linux system, you can identify angel processes by matching their process IDs with the owner IDs of the remaining ndbd processes in the output of ps excuted on host beta of the example cluster, as shown here, with the relevant process IDs shown in emphasized text:

shell> ps -ef | grep ndbd
jon       2023     1  0 18:46 ?        00:00:00 ./ndbd -c alpha
jon    2024  2023  1 18:46 ?        00:00:00 ./ndbd -c alpha
jon    2124  1819  0 18:46 pts/2    00:00:00 grep --color=auto ndbd

Use the kill command to terminate the process with the indicated process ID, like this:

shell> kill -9 2023

Verify that the angel process has been killed, and that only one of the two original ndbd processes remain, by issuing ps again, as shown here:

shell> ps -ef | grep ndbd
jon    2024     1  1 18:46 ?        00:00:01 ./ndbd -c alpha
jon    2150  1819  0 18:47 pts/2    00:00:00 grep --color=auto ndbd

Now repeat this process from a login shell on host gamma, as shown here:

shell> ps -ef | grep ndbd
jon       2066     1  0 18:46 ?        00:00:00 ./ndbd -c alpha
jon    2067  2066  1 18:46 ?        00:00:00 ./ndbd -c alpha
jon    3712  1704  0 18:46 pts/2    00:00:00 grep --color=auto ndbd

shell> kill -9 2066

shell> ps -ef | grep ndbd
jon    2067     1  1 18:46 ?        00:00:01 ./ndbd -c alpha
jon    2150  1819  0 18:47 pts/2    00:00:00 grep --color=auto ndbd

The wild cluster's data nodes are now ready for migration.

Kill and restart each management node process. When restarting ndb_mgmd, its configuration cache must be disabled; since this is enabled by default, you must start the management server with --config-cache=false, in addition to any other options that it was previously started with.

On Linux, we can once again use ps to obtain the information we need to accomplish this, this time in a shell on host alpha:

shell> ps -ef | grep ndb_mgmd
jon   16005     1  1 18:46 ?        00:00:09 ./ndb_mgmd -f /etc/mysql-cluster/config.ini
jon   16401  1819  0 18:58 pts/2    00:00:00 grep --color=auto ndb_mgmd

The process ID is 16005, and the management node was started with the -f option (the short form for --config-file). First, terminate the management using kill, as shown here, with the process ID obtained from ps previously:

shell> kill -9 16005

Verify that the management node process was killed, like this:

shell> ps -ef | grep ndb_mgmd
jon    16532  1819  0 19:03 pts/2    00:00:00 grep --color=auto ndb_mgmd

Now restart the management node as described previously, with the same options that it was started with previously, and with the configuration cache disabled. Change to the directory where ndb_mgmd is located, and restart it, like this:

shell> ./ndb_mgmd -f /etc/mysql-cluster/config.ini --config-cache=false
MySQL Cluster Management Server mysql-5.6.24-ndb-7.4.6
2013-12-06 19:16:08 [MgmtSrvr] INFO -- Skipping check of config directory since
config cache is disabled.

Verify that the process is running as expected, using ps:

shell> ps -ef | grep ndb_mgmd
jon    17066  1  1 19:16 ?      00:00:01 ./ndb_mgmd -f
/etc/mysql-cluster/config.ini --config-cache=false
jon    17311  1819  0 19:17 pts/2  00:00:00 grep --color=auto ndb_mgmd

The management node is now ready for migration.

Any cluster processes that are under the control of a system boot process management facility, such as /etc/init.d on Linux systems or the Services Manager on Windows platforms, should be removed from this facility's control. Consult your system operating documentation for information about how to do this. Be sure not to stop any running cluster processes in the course of doing so.

It is highly recommended that you take a complete backup of the “wild” cluster before proceeding any further, using the ndb_mgm client's START BACKUP command:

ndb_mgm> START BACKUP
Waiting for completed, this may take several minutes
Node 5: Backup 1 started from node 1
Node 5: Backup 1 started from node 1 completed
 StartGCP: 1338 StopGCP: 20134
 #Records: 205044 #LogRecords: 10112
 Data: 492807474 bytes Log: 317805 bytes

It may require some time for the backup to complete, depending on the size of the cluster's data and logs. For START BACKUP command options and additional information, see Using The NDB Cluster Management Client to Create a Backup.

You must verify that each process in the “wild” cluster has a valid PID file. For purposes of this discussion, a valid PID file has the following characteristics:

To check the PID file for the management node process, log into the system shell on host alpha, and change to the management node's data directory. If this is not specified, the PID file should be created in the same directory that ndb_mgmd runs in; change to this directory instead. Then check to see whether the PID file is present using your system's tools for doing this. On Linux, you can use the command shown here:

shell> ls ndb_1*
ndb_1_cluster.log  ndb_1_out.log  ndb_1.pid

Check the content of the matching .pid file using a pager or text editor. We use more for this purpose here:

shell> more ndb_1.pid
17066

The number shown should match the ndb_mgmd process ID. We can check this on Linux as before, using ps:

shell> ps -ef | grep ndb_mgmd
jon    17066  1  1 19:16 ?      00:00:01 ./ndb_mgmd -f /etc/mysql-cluster/config.ini --config-cache=false
jon    17942  1819  0 19:17 pts/2  00:00:00 grep --color=auto ndb_mgmd

The management node PID file satisfies the requirements listed at the beginning of this section. Next, we check the PID files for the data nodes, on hosts beta and gamma. Log into a system shell on beta, then obtain the process ID of the ndbd process on this host, as shown here:

shell> ps -ef | grep ndbd
jon    2024     1  1 18:46 ?        00:00:01 ./ndbd -c alpha
jon    2150  1819  0 18:47 pts/2    00:00:00 grep --color=auto ndbd

We observed earlier (see Section 3.5.2.1, “Creating and Configuring the Target Cluster”) that this node's node ID is 5 and that its DataDir is /var/lib/mysql-cluster. Check in this directory for the presence of a file named ndb_5.pid:

shell> ls /var/lib/mysql-cluster/ndb_5.pid
ndb_5.pid

Now check the content of this file and make certain that it contains the process ID 2024 on the first line and no other content, like this:

shell> more /var/lib/mysql-cluster/ndb_5.pid
2024

Similarly, we locate and check the content of the PID file for the remaining data node (node ID 6, data directory /var/lib/mysql-cluster/) on host gamma:

shell> ps -ef | grep ndbd
jon    2067     1  1 18:46 ?        00:00:01 ./ndbd -c alpha
jon    2150  1819  0 18:47 pts/2    00:00:00 grep --color=auto ndbd

shell> ls /var/lib/mysql-cluster/ndb_6.pid
ndb_6.pid

shell> more /var/lib/mysql-cluster/ndb_6.pid
2067

The PID file for this data node also meets our requirements, so we are now ready to proceed to the mysqld binary running on host delta. We handle the PID file for this process in the next step.

If a given process does not have a valid PID file, you must create one for it, or, in some cases, modify the existing one. This is most likely to be a concern when checking PID files for mysqld processes, due to the fact that the MySQL Server is customarily started using the startup script mysqld_safe, which can start the mysqld binary with any number of default options, including the --pid-file option. We see that is the case when we check on host delta for the running mysqld process there (emphasized text):

shell> ps -ef | grep mysqld
jon   8782  8520  0 10:30 pts/3  00:00:00 /bin/sh ./mysqld_safe --ndbcluster
--ndb-connectstring=alpha
jon   8893  8782  1 10:30 pts/3  00:00:00 /usr/local/mysql/bin/mysqld
--basedir=/usr/local/mysql --datadir=/usr/local/mysql/data
--plugin-dir=/usr/local/mysql/lib/plugin --ndbcluster --ndb-connectstring=alpha
--log-error=/usr/local/mysql/data/delta.err --pid-file=/usr/local/mysql/data/delta.pid
jon       8947  8520  0 10:30 pts/3    00:00:00 grep --color=auto mysqld

shell> more /usr/local/mysql/data/delta.pid
8893

The PID for the SQL node is in an acceptable location (the data directory) and has the correct content, but has the wrong name.

You can create a correct PID file in either of two locations—in the process data directory, or in the directory mcm_dir/clusters/cluster name/pid/ on the same host as the process, where mcm_dir is the MySQL Cluster Manager installation directory, and cluster_name is the name of the cluster. In this case, since the existing PID file is otherwise correct, it is probably easiest just to copy it to a correctly named file in the same directory incorporating the node ID (100), like this:

shell> cp /usr/local/mysql/data/delta.pid /usr/local/mysql/data/ndb_100.pid

Another alternative is to create and write a completely new PID file to the proper location in the MySQL Cluster Manager installation directory, as shown here:

shell> echo '8893' > /opt/mcm-1.3.6/clusters/newcluster/pid/ndb_100.pid

shell> more /opt/mcm-1.3.6/clusters/newcluster/pid/ndb_100.pid
8893

ndbapi processes running under MySQL Cluster Manager do not require PID files, so we have completed this step of the import, and we should be ready for a test or “dry run” of the migration. We perform this test in the next step.

Perform a test run of the proposed import using import cluster with the --dryrun option. When this option is used, MySQL Cluster Manager checks for mismatched configuration attributes, missing or invalid processes or hosts, missing or invalid PID files, and other errors, and warns of any it finds, but does not actually perform any migration of processes or data.

mcm> import cluster --dryrun newcluster;
ERROR 5302 (00MGR): No access for user mcmd to mysqld 100 in cluster newcluster.
Please verify user access and grants adhere to documented requirements.

We omitted a crucial step earlier: we apparently neglected to create the mcmd superuser account needed on all SQL nodes in the “wild” cluster to bring them under control of MySQL Cluster Manager. In this case, there is only one SQL node, running on delta. Log into this SQL node as the MySQL root user, and create the mcmd account in the mysql client, as shown here:

shell> ./mysql -uroot -p
Enter password: ************
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.6.24-ndb-7.4.6 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'mcmd'@'localhost' IDENTIFIED BY 'super';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.*
    ->     TO 'mcmd'@'localhost' IDENTIFIED BY 'super'
    ->     WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW GRANTS FOR 'mcmd'@'localhost'\G
*************************** 1. row ***************************
Grants for mcmd@localhost: GRANT ALL PRIVILEGES ON *.* TO 'mcmd'@'localhost'
IDENTIFIED BY PASSWORD '*F85A86E6F55A370C1A115F696A9AD71A7869DB81' WITH GRANT OPTION
1 row in set (0.00 sec)

Having corrected this issue on the SQL node, we repeat the dry run in the mcm client:

mcm> import cluster --dryrun newcluster;
ERROR 5310 (00MGR): Process ndb_mgmd 50 reported 6 processes, while 5 processes
are configured for cluster newcluster

This error means that there are one or more cluster processes not accounted for the configuration for the target cluster. Checking the contents of the file /etc/mysql-cluster/config.ini on host alpha, we see that we overlooked a section in it earlier. This section is shown here:

[mysqld]
NodeId=102

To address this discrepancy, we need to add another “free” ndbapi process to newcluster, which we can do by executing the following add process command in the mcm client:

mcm> add process -R ndbapi:102@* newcluster;
+----------------------------+
| Command result             |
+----------------------------+
| Process added successfully |
+----------------------------+
1 row in set (0.38 sec)

You can verify this by checking the output of show status -r command, as shown here:

mcm> show status -r newcluster;
+--------+----------+-------+--------+-----------+------------+
| NodeId | Process  | Host  | Status | Nodegroup | Package    |
+--------+----------+-------+--------+-----------+------------+
| 50     | ndb_mgmd | alpha | import |           | newpackage |
| 5      | ndbd     | beta  | import | n/a       | newpackage |
| 6      | ndbd     | gamma | import | n/a       | newpackage |
| 100    | mysqld   | delta | import |           | newpackage |
| 101    | ndbapi   | *     | import |           |            |
| 102    | ndbapi   | *     | import |           |            |
+--------+----------+-------+--------+-----------+------------+
6 rows in set (0.11 sec)

Now we can run another test, using import cluster with the --dryrun as we did previously:

mcm> import cluster --dryrun newcluster;

Continue to correct any errors or other discrepancies found using --dryrun, repeating the dry run shown in the previous step to ensure that no errors were missed. The following list contains some common errors you may encounter, and their likely causes:

When import cluster --dryrun no longer warns of any errors, you can perform the import with the import cluster command, this time omitting the --dryrun option.