Chapter 1 Introduction to MySQL Enterprise Backup

Chapter 1 Introduction to MySQL Enterprise Backup
Prev		Next

Table of Contents

1.1 Types of Backups
1.2 The mysqlbackup Command
1.3 Overview of Backup Performance and Capacity Considerations
1.4 Files that Are Backed Up
1.5 Overview of Restoring a Database

The MySQL Enterprise Backup product performs backup operations for MySQL data. It can back up all kinds of MySQL tables. It has special optimizations for fast and convenient backups of InnoDB tables. Because of the speed of InnoDB backups, and the reliability and scalability features of InnoDB tables, we recommend that you use InnoDB tables for your most important data.

This book describes the best practices regarding MySQL backups and documents how to use MySQL Enterprise Backup features to implement these practices. This book teaches you:

Why backups are important.
The files that make up a MySQL database and the roles they play.
How to keep the database running during a backup.
How to minimize the time, CPU overhead, and storage overhead for a backup job. Often, minimizing one of these aspects increases another.
How to restore your data when disaster strikes. You learn how to verify backups and practice recovery, so that you can stay calm and confident under pressure.
Other ways to use backup data for day-to-day administration and in deploying new servers.

1.1 Types of Backups

The various kinds of backup techniques are classified on a scale ranging from hot (the most desirable) to cold (the most disruptive). Your goal is to keep the database system, and associated applications and web sites, operating and responsive while the backup is in progress.

Hot backups are performed while the database is running. This type of backup does not block normal database operations. It captures even changes that occur while the backup is happening. For these reasons, hot backups are desirable when your database “grows up”: when the data is large enough that the backup takes significant time, and when your data is important enough to your business so that you must capture every last change, without taking your application, web site, or web service offline.

MySQL Enterprise Backup does a hot backup of all InnoDB tables. MyISAM and other non-InnoDB tables are backed up last, using the warm backup technique: the database continues to run, but the system is in a read-only state during that phase of the backup.

You can also perform cold backups while the database is stopped. To avoid service disruption, you would typically perform such a backup from a replication slave, which can be stopped without taking down the entire application or web site.

Points to Remember

To back up as much data as possible during the hot backup phase, you can designate InnoDB as the default storage engine for new tables, or convert existing tables to use the InnoDB storage engine. (In MySQL 5.5 and higher, InnoDB is now the default storage engine for new tables.)

During hot and warm backups, information about the structure of the database is retrieved automatically through a database connection. For a cold backup, you must specify file locations through configuration files or command-line options.

1.2 The mysqlbackup Command

When using the MySQL Enterprise Backup product, you primarily work with the mysqlbackup command. Based on the options you specify, this command performs all the different types of backup operations, and restore operations too. mysqlbackup can do other things that you would otherwise code into your own backup scripts, such as creating a timestamped subdirectory for each backup, compressing the backup data, and packing the backup data into a single file for easy transfer to another server.

For usage information about mysqlbackup features, see Chapter 3, Backing Up a Database Server. For option syntax, see Chapter 5, mysqlbackup Command Reference.

1.3 Overview of Backup Performance and Capacity Considerations

In your backup strategy, performance and storage space are key aspects. You want the backup to complete quickly, with little CPU overhead on the database server. You also want the backup data to be compact, so you can keep multiple backups on hand to restore at a moment's notice. Transferring the backup data to a different system should be quick and convenient. All of these aspects are controlled by options of the mysqlbackup command.

Sometimes you must balance the different kinds of overhead -- CPU cycles, storage space, and network traffic. Always be aware how much time it takes to restore the data during planned maintenance or when disaster strikes. For example, here are factors to consider for some of the key MySQL Enterprise Backup features:

Parallel backups are the default in MySQL Enterprise Backup 3.8, a major performance improvement over earlier MySQL Enterprise Backup releases. The read, process and write are the primary sub-operations of all MEB operations. For example, in a backup operation, MySQL Enterprise Backup first reads the data from the disk, then processes this data, writes the data to disk, and reads the data again for verification. MySQL Enterprise Backup ensures that these sub-operations are independent of each other and run in parallel to gain performance improvement. Read, process and write sub-operations are performed in parallel using multiple threads of the same kind: multiple read threads, multiple process threads, and multiple write threads, resulting in better performance. The performance improvement is typically greater when RAID arrays are used as both source and target devices, and for compressed backups which can use more CPU cycles in parallel.
Parallel backup employs block-level parallelism, using blocks of 16MB. Different threads can read, process, and write different 16MB chunks within a single file. Parallel backup improves the performance of operations whether the instance contains a single huge system tablespace, or many smaller tablespaces (represented by .ibd files created in the innodb_file_per_table mode).
Incremental backups are faster than full backups, save storage space on the database server, and save on network traffic to transfer the backup data on a different server. Incremental backup requires additional processing to make the backup ready to restore, which you can perform on a different system to minimize CPU overhead on the database server.
Compressed backups save on storage space for InnoDB tables, and network traffic to transfer the backup data on a different server. They do impose more CPU overhead than uncompressed backups. During restore, you need the compressed and uncompressed data at the same time, so take into account this additional storage space and the time to uncompress the data.
In addition to compressing data within InnoDB tables, compressed backups also skip unused space within InnoDB tablespace files. Uncompressed backups include this unused space.
When space is limited, or you have a storage device such as tape that is cheap, large, but also slow, the performance and space considerations are different. Rather than aiming for the fastest possible backup, you want to avoid storing an intermediate copy of the backup data on the database server. MySQL Enterprise Backup can produce a single-file backup and stream that file directly to the other server or device. Since the backup data is never saved to the local system, you avoid the space overhead on the database server. You also avoid the performance overhead of saving a set of backup files and then bundling them into an archive for transport to another server or storage device. For details, see Section 3.3.5.1, “Streaming the Backup Data to Another Device or Server”.
When streaming backup data to tape, you typically do not compress the backup, because the CPU overhead on the database server to do the compression is more expensive than the additional storage space on the tape device. When streaming backup data to another server, you might compress on the original server or the destination server depending on which server has more spare CPU capacity and how much network traffic the compression could save. Or, you might leave the backup data uncompressed on the destination server so that it is ready to be restored on short notice.

For disaster recovery, when speed to restore the data is critical, you might prefer to have critical backup data already prepared and uncompressed, so that the restore operation involves as few steps as possible.

It is during a disaster recovery that speed is most critical. For example, although a logical backup performed with the mysqldump command might take about the same time as a physical backup with the MySQL Enterprise Backup product (at least for a small database), the MySQL Enterprise Backup restore operation is typically faster. Copying the actual data files back to the data directory skips the overhead of inserting rows and updating indexes that comes from replaying the SQL statements from mysqldump output.

To minimize any impact on server performance on Linux and Unix systems, MySQL Enterprise Backup writes the backup data without storing it in the operating system's disk cache, by using the posix_fadvise() system call. This technique minimizes any slowdown following the backup operation, by preventing frequently accessed data from being flushed from the disk cache by the large one-time read operation for the backup data.

For more on techniques and tradeoffs involving backup and restore performance, see Chapter 7, Performance Considerations for MySQL Enterprise Backup.

1.4 Files that Are Backed Up

DBA and development work typically involves logical structures such as tables, rows, columns, the data dictionary, and so on. For backups, you must understand the physical details of how these structures are represented by files.

Table 1.1 Files in a MySQL Enterprise Backup Output Directory

File Name, Pattern, or Extension	Relation to Original Data Files	Notes
`ibdata*`	The InnoDB system tablespace, containing multiple InnoDB tables and associated indexes.	Because the original files might change while the backup is in progress, the `apply-log` step applies the same changes to the corresponding backup files.
`*.ibd`	InnoDB file-per-table tablespaces, each containing a single InnoDB table and associated indexes.	Used for tables created under the `innodb_file_per_table`. Because the original files might change while the backup is in progress, the apply-log step applies the same changes to the corresponding backup files.
`*.ibz`	Compressed form of InnoDB data files from the MySQL data directory.	Produced instead of `.ibd` files in a compressed backup. The `ibdata*` files representing the InnoDB system tablespace also receive this extension in a compressed backup. The `.ibz` files are uncompressed for the apply-log step.
`*.frm`	Hold metadata about all MySQL tables.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.MYD`	MyISAM table data.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.MYI`	MyISAM index data.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.CSM`	Metadata for CSV tables.	These files are copied without changes. The `backup_history` and `backup_progress` tables created by mysqlbackup use the CSV format, so the backup always includes some files with this extension.
`*.CSV`	Data for CSV tables.	These files are copied without changes. The `backup_history` and `backup_progress` tables created by mysqlbackup use the CSV format, so the backup always includes some files with this extension.
`*.MRG`	MERGE storage engine references to other tables.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.TRG`	Trigger parameters.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.TRN`	Trigger namespace information.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.opt`	Database configuration information.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.par`	Definitions for partitioned tables.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.ARM`	Archive storage engine metadata.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`*.ARZ`	Archive storage engine data.	The database is put into a read-only state while these files are copied. These files are copied without changes.
`backup-my.cnf`	Records the configuration parameters that specify the layout of the MySQL data files.	Used in restore operations to reproduce the same layout as when the backup was taken.
`ibbackup_logfile`	A condensed version of the `ib_logfile*` files from the MySQL data directory.	The InnoDB log files (`ib_logfile`) are fixed-size files that are continuously updated during database operation. For backup purposes, only the changes that are committed while the backup is in progress are needed. These changes are recorded in `ibbackup_logfile`, and used to re-create the `ib_logfile` files during the apply-log phase.
`ibbackup_redo_log_only`	Used instead of `ibbackup_logfile` for incremental backups taken with the `--incremental-with-redo-log-only` option.
`ib_logfile*`	Created in the backup directory during the apply-log phase after the initial backup.	These files are not copied from the original data directory, but rather re-created in the backup directory during the apply-log phase after the initial backup, using the changes recorded in the `ibbackup_logfile` file.
`*.bl`	Renamed version of each .isl file from the backed-up server.	A `.isl` file is created when you specify the location of an `InnoDB` table using the syntax `CREATE TABLE ... DATA DIRECTORY = ...`, to act like a symbolic link pointing to the tablespace file. (See Creating a File-Per-Table Tablespace Outside the Data Directory for details.) The `.bl` files might or might not be turned back into `.isl` files during the `copy-back` operation. If the specified directory does not exist on the server where the backup is restored, the mysqlbackup command attempts to create it. If the directory cannot be created, the restore operation fails. Thus, if you use the `DATA DIRECTORY` clause to put tables in different locations, and restore to a server with a different file structure where the corresponding directories cannot be created, edit the `.bl` files before restoring to point to directories that do exist on the destination server.
Timestamped directory, such as `2011-05-26_13-42-02`	Created by the `--with-timestamp` option. All the backup files go inside this subdirectory.	Use the `--with-timestamp` option whenever you intend to keep more than one set of backup data available under the same main backup directory.
`datadir` directory	A subdirectory that stores all the data files and database subdirectories from the original MySQL instance.	Created under the backup directory by the mysqlbackup command.
binary log files	Binary log files from the server, which are included in a backup by default (except when the backup is created with the `--use-tts` option). They allow a snapshot of the server to be taken, so a server can be cloned to its exact state. Using a full backup as a basis, the binary log files that are included with an incremental backup can be used for a point-in-time recovery (PITR), which restores a database to its state at a certain point in time after the last full backup. See Section 4.3, “Point-in-Time Recovery from a Hot Backup” for details.	Saved under the `datadir` directory under the backup directory. Use the `--skip-binlog` option to exclude binary logs in the backup. For MySQL 5.5 and earlier, as well as all offline backups, use the `--log-bin-index` option to specify the absolute path of the index file on the MySQL server that lists all the used binary log files, if it is different from the default value of the option, for mysqlbackup to find the binary log files and include them in the backups. Note Due to some known issues, users should always use the `--skip-binlog` option when creating offline backups, or when creating an incremental backup that is based on a full backup created with the `--no-locking` option. See Appendix A, MySQL Enterprise Backup Limitations for details.
relay log files	Relay log files from a slave server, which are included in a backup of a slave server by default (except when the backup is created with the `--use-tts` option). Their inclusion saves the time and resources required for fetching the relay logs from the master when the slave is being restored.	Saved under the `datadir` directory under the backup directory. Use the `--skip-relaylog` option to exclude relay logs in the backup. For offline backup, use the `--relay-log-index` option to specify the absolute path of the index file on the MySQL server that lists all the used relay log files, if it is different from the default value of the option, for mysqlbackup to find the relay log files and include them in the backups.
slave status log files	Usually named `master.info` and `relay-log.info`, they are included by default in a backup of a slave database in a replication setup. See Slave Status Logs, for details.	Saved under the `datadir` directory under the backup directory. For an offline backup, use the`--master-info-file` and `--relaylog-info-file` options to specify the absolute paths of the information files, if they are different from the default values of the options, for mysqlbackup to find those files and include them in the backups.
`image file`	A single-file backup produced by the `backup-to-image` option, with a name specified by the `--backup-image` option.	If your backup data directory consists only of zero-byte files, with a single giant data file in the top-level directory, you have a single-file backup. You can move the image file without losing or damaging the contents inside it, then unpack it with the mysqlbackup command using the `extract` option and specifying the same image name with the `--backup-image` option. Although some extra files such as `backup-my.cnf` and the `meta` subdirectory are present in the backup directory, these files are also included in the image file and do not need to be moved along with it.
Any other files in the database subdirectories under the `datadir` directory (that is, under `backup-dir/datadir/data-base-name`)	Copied from the database subdirectories under the MySQL data directory.	By default, any unrecognized files in the database subdirectories under the MySQL data directory are copied to the backup. To omit such files, specify the `--only-known-file-types` option.
`meta` directory	A subdirectory that stores files with metadata about the backup.	Created under the backup directory by the mysqlbackup command. All files listed below go inside the `meta` subdirectory.
`backup_variables.txt`	Holds important information about the backup. For use by the mysqlbackup command only. Prior to MySQL Enterprise Backup 3.6, this information was in a file named `ibbackup_binlog_info`.	The mysqlbackup command consults and possibly updates this file during operations after the initial backup, such as the apply-log phase or the restore phase.
`image_files.xml`	Contains the list of all the files (except itself) that are present in the single-file backup produced by the `backup-to-image` or `backup-dir-to-image` options. For details about this file, see Section 11.4, “Using the MySQL Enterprise Backup Manifest”.	This file is not modified at any stage once generated.
`backup_create.xml`	Lists the command line arguments and environment in which the backup was created. For details about this file, see Section 11.4, “Using the MySQL Enterprise Backup Manifest”.	This file is not modified once it is created. You can prevent this file from being generated by specifying the `--disable-manifest` option.
`backup_content.xml`	Essential metadata for the files and database definitions of the backup data. It also contains details of all the plugins defined on the backed-up server, by which users should make sure the same plugins are defined in the same manner on the target server for restoration. For details about this file, see Section 11.4, “Using the MySQL Enterprise Backup Manifest”.	This file is not modified once created. You can prevent this file from being generated by specifying the `--disable-manifest` option.
`comments.txt`	Produced by the `--comments` or `--comments-file` option.	The comments are specified by you to document the purpose or special considerations for this backup job.
`gtid_executed.sql`	Signifies the backup came from a server with GTIDs enabled.	GTIDs are a replication feature in MySQL 5.6 and higher. See Replication with Global Transaction Identifiers for details. When you back up a server with GTIDs enabled, the file `gtid_executed.sql` is created in the backup directory. Edit and execute this file after restoring the backup data on a slave server.
`server-my.cnf`	Contains values of the backed-up server's global variables that are set to non-default values. Use this file or `server-all.cnf` to start the target server for restoration.	During a `copy-back` or `copy-back-and-apply-log` operation, the server repository options values (e.g., `--datadir`, `--innodb_data_home_dir`, etc.) in the file are modified if the command makes changes to them through the command options. However, during an `--apply-incremental-backup` operation, the values already saved in the file take precedence and they are not modified by the option values supplied through the command. Warning When using the file to restart the target server, change parameters like `--tmpdir`, `--general-log`, etc., and any global variable that uses an absolute filepath to avoid the accidental usage of the wrong file locations by the target server.
`server-all.cnf`	Contains values of all the global variables of the backed-up server. Use this file or `server-my.cnf` to start the target server for restoration.	During a `copy-back` or `copy-back-and-apply-log` operation, the server repository options values (e.g., `--datadir`, `--innodb_data_home_dir`, etc.) in the file are modified if the command makes changes to them through the command options. However, during an `--apply-incremental-backup` operation, the values already saved in the file take precedence and they are not modified by the option values supplied through the command. Warning When using the file to restart the target server, change parameters like `--tmpdir`, `--general-log`, etc., and any global variable that uses an absolute filepath to avoid the accidental usage of the wrong file locations by the target server.

InnoDB Data

Data managed by the InnoDB storage engine is always backed up. The primary InnoDB-related data files that are backed up include the ibdata* files that represent the system tablespace and possibly the data for some user tables; any .ibd files, containing data from user tables created with the file-per-table setting enabled; data extracted from the ib_logfile* files (the redo log information representing changes that occur while the backup is running), which is stored in a new backup file ibbackup_logfile.

If you use the compressed backup feature, the .ibd files are renamed in their compressed form to .ibz files.

The files, as they are originally copied, form a raw backup that requires further processing before it is ready to be restored. You then run the apply step, which updates the backup files based on the changes recorded in the ibbackup_logfile file, producing a prepared backup. At this point, the backup data corresponds to a single point in time. The files are now ready to be restored to their original location, or for some other use, such as testing, reporting, or deployment as a replication slave.

To restore InnoDB tables to their original state, you must also have the corresponding .frm files along with the backup data. Otherwise, the table definitions could be missing or outdated if someone has run ALTER TABLE or DROP TABLE statements since the backup. By default, the mysqlbackup command automatically copies the .frm files during a backup operation and restores the files during a restore operation.

Data from MyISAM and Other Storage Engines

The mysqlbackup command can also back up the .MYD files, .MYI files, and associated .frm files for MyISAM tables. The same applies to files with other extensions, as shown in this list.

Note

While MySQL Enterprise Backup can back up non-InnoDB data (like MYISAM tables), the MySQL server to be backed up must support InnoDB (i.e., the backup process will fail if the server was started up with the --innodb=OFF or --skip-innodb option), and the server must contain at least one InnoDB table.

MyISAM tables and these other types of files cannot be backed up in the same non-blocking way as InnoDB tables can. This phase is a warm backup: changes to these tables are prevented while they are being backed up, possibly making the database unresponsive for a time, but no shutdown is required during the backup.

Note

To avoid concurrency issues during backups of busy databases, you can use the --only-innodb or --only-innodb-with-frm option to back up only InnoDB tables and associated data.

Generated Files Included in the Backup

The backup data includes some new files that are produced during the backup process. These files are used to control later tasks such as verifying and restoring the backup data. The files generated during the backup process include:

backup-my.cnf: Records the crucial configuration parameters that apply to the backup. These parameter values are used during a restore operation, so that the original values are used regardless of changes to your my.cnf file in the meantime.
meta/backup_create.xml: Lists the command line arguments and environment in which the backup was created.
meta/backup_content.xml: Essential metadata for the files and database definitions of the backup data.
server-my.cnf: Contains values of the backed-up server's global variables that are set to non-default values.
server-all.cnf: Contains values of all the global variables of the backed-up server.

For details about all the files in the backup directory, see Table 1.1, “Files in a MySQL Enterprise Backup Output Directory”.

Single-File Backups

Depending on your workflow, you might perform a single-file backup rather than the typical backup that produces a separate file for every file in the original instance. The single-file format is easier to transfer to a different system, compress and uncompress, and ensure that no backed-up files are deleted later by mistake. It is just as fast as a multi-file backup to do a full restore; restoring individual files can be slower than in a multi-file backup. For instructions, see Section 3.3.5, “Making a Single-File Backup”.

1.5 Overview of Restoring a Database

To initiate the restore process, you run the mysqlbackup command with the copy-back or the copy-back-and-apply-log subcommand. You can restore all the data for a MySQL server: multiple databases, each containing multiple tables. Or, you can restore selected databases, tables, or both.

To repair a problem such as data corruption, you restore the data back to its original location on the original server machine. You might restore to a different server machine or a different location to set up a new replication slave with the data from a master server, or to clone a database for reporting purposes.

See Chapter 4, Recovering or Restoring a Database for instructions on restoring databases.

Prev	Up	Next
Part I Getting Started with MySQL Enterprise Backup	Home	Chapter 2 Installing MySQL Enterprise Backup