Chapter 6 ndbmemcache—Memcache API for NDB Cluster

Memcached is a distributed in-memory caching server using a simple text-based protocol, commonly used for key-value data stores, with clients available for many platforms and programming languages. The most recent release of the memcached server is available from memcached.org.

The Memcache API for NDB Cluster is available beginning with NDB 7.2.2. This API is implemented as a loadable storage engine for memcached version 1.6 and later, which employs a storage engine architecture. This API can be used to provide a persistent NDB Cluster data store which is accessible employing the memcache protocol. It is also possible for the memcached server to provide a strictly defined interface to existing NDB Cluster tables such that an administrator can control exactly which tables and columns are referenced by particular memcache keys and values, and which operations are allowed on these keys and values.

The standard memcached caching engine is included in the NDB Cluster distribution. Each memcache server, in addition to providing direct access to data stored in NDB Cluster, is able to cache data locally and serve (some) requests from this local cache. As with table and column mappings, cache policies are configurable based on a prefix of a memcache key.

Support for the Memcache API is built automatically using the memcached and libevent sources included in the NDB Cluster sources when compiling NDB 7.2.3 or later from source. By default, make install places the memcached binary in the NDB Cluster installation bin directory, and the ndbmemcache engine shared object file ndb_engine.so in the installation lib directory.

You can disable use of the bundled memcached when building ndbmemcache, by using -DWITH_BUNDLED_MEMCACHED=OFF; you can instead use your own system's memcached server and sources, installed in path, with -DWITH_BUNDLED_MEMCACHED=OFF -DMEMCACHED_HOME=path. You can also cause your system's version of libevent to be used, rather than the version bundled with NDB Cluster, by using the -DWITH_BUNDLED_LIBEVENT=OFF option.

For additional information about CMake options relating to ndbmemcache support, see Options for Compiling NDB Cluster.

For general information about building NDB Cluster, see Building NDB Cluster from Source on Linux, and Compiling and Installing NDB Cluster from Source on Windows. For information about building MySQL Server from source, see Installing MySQL from Source, as well as MySQL Source-Configuration Options.

The following list contains memcached command line options that are of particular interest or usefulness when working with ndbmemcache.

For general information memcached command line options, see the documentation at http://code.google.com/p/memcached/wiki/NewStart.

NDB memcache engine configuration options.  The NDB engine supports the following configuration options for use with memcache -e (see Section 6.3, “memcached command line options”):

Initial Configuration.  When a the NDB engine starts up, its most important command-line arguments are the cluster connection string and server role. The connection string is used to connect to a particular cluster, called the primary cluster, which contains a configuration schema. The tables in the configuration schema are read to retrieve a set of key-prefix mappings for the given server role (see the ndbmemcache configuration schema). Those mappings instruct the server how to respond to memcache operations on particular keys, based on the leftmost part of the key. For instance, they may specify that data is stored in particular columns of a certain table. This table may be stored in the same cluster as the configuration schema, or in a different cluster. A memcache server may have connections to several different clusters, and many memcache servers may connect to a single cluster but with a variety of roles.

The ndbmemcache configuration schema.  When the memcache NDB engine starts up, it connects to a cluster, and looks for the ndbmemcache configuration schema there. If the schema is not found, it shuts down.

The schema is described (with full comments) in the file ndb_memcache_metadata.sql

The main concept of the schema is a key-prefix mapping. This takes a prefix of a memcache key and maps it to a specific container table, on a particular cluster, with a particular cache policy.

A server role is defined as a set of key-prefix mappings that a memcached server will implement.

Whenever a memcached server is started with a particular server role (from the command-line arguments), that server role must exist in the ndbmemcache.server_roles table.

The following table lists table names and descriptions for tables that belong to the ndbmemcache configuration schema.

Cache policies.  There are four policy types: get_policy, set_policy, delete_policy, and flush_from_db. These are described in the following paragraphs.

get_policy determines how the memcached server interprets GET commands. Possible values and their meanings are shown in the following list:

The set_policy determines how the memcached server interprets SET, INSERT, and REPLACE commands. Possible set_policy values and their meanings are listed here:

delete_policy describes how the memcached server interprets DELETE commands. It can take on the values shown and described in the following list:

flush_from_db determines how the memcached server interprets a FLUSH_ALL command with regard to data stored in the NDB Cluster database, as shown here:

containers table columns.  The columns in the containers table are described in the following list:

Key mappings. 

The following table lists table names and descriptions for non-configuration ndbmemcache logging and container tables.

Predefined configuration objects

Predefined clusters.  A single ndb_cluster record is predefined, referring to the primary cluster (the one where configuration data is stored) as cluster id 0. Id 0 should always be reserved for the primary cluster.

Predefined cache policies

Predefined containers

Predefined memcache server roles and their key prefixes

Configuration versioning and upgrade.  The configuration schema is versioned, and the version number is stored in the ndbmemcache.meta table. The NDB Engine begins the configuration process by reading the schema version number from this table. As a rule, newer versions of the NDB engine will remain compatible with older versions of the configuration schema.

STABILITY NOTE: consider this section "unstable" & subject to change

Performance Tuning.  Two parameters are used to tune performance of the NDB memcache engine. The parameters are stored in the configuration schema: the "usec_rtt" value of a particular cluster, and the "max_tps" value of a memcache server role. These values are currently used in two ways: to configure the number of connections to each cluster, and to configure a particular fixed number of concurrent operations supported from each connection.

Autotuning.  Autotuning uses an estimated round trip time between cluster data nodes and a target rate of throughput to determine the ideal number of cluster connections and transactions per connection for a given workload. Autotuning parameters are described in the next few paragraphs.

These values are used, as described in the next few paragraphs, to calculate an optimum number of cluster connections with a given transactions-per-second capacity..

Number of cluster connections.  The NDB Engine scheduler attempts to open 1 cluster connection per 50000 transactions per second (TPS). This behavior can be overridden by using a scheduler configuration string (see Section 6.4, “NDB Engine Configuration”.) If the scheduler fails to open a second or subsequent connection to a cluster—for example, because a node id is not available—this is not a fatal error; it will run with only the connections actually opened.

Number of transactions per connection.  We assume that a transaction takes 5 times the cluster round trip time to complete. We can obtain the total number of in-flight transactions by dividing the server's max_tps by 5 * rtt (in seconds). These in-flight transaction objects are evenly distributed among the cluster connections.

Tuning example.  The following example starts with the default values usec_rtt = 250 and max_tps = 100000, and assumes a memcached server with 4 worker threads.

Online reconfiguration.  It is possible to reconfigure the key-prefix mappings of a running NDB engine without restarting it. This is done by committing a change to the configuration schema, and then updating the update_timestamp column of a particular server role in the memcache server roles table. The updating of the timestamp causes an event trigger to fire, so that the memcache server receives notification of the event.

Online reconfiguration can be disabled by using the -e reconf=false option on the command line.

Online reconfiguration can be used to connect to new clusters and to create new key-prefix mappings. However, it cannot be used to reset autotuning values on existing connections.

Online reconfiguration is a risky operation that could result in memcache server crashes or data corruption, and is used extensively in the mysql test suite. However, it is not recommended for reconfiguring a production server under load.

The stats reconf command can be run before and after online reconfiguration to verify that the version number of the running configuration has increased. Verification of reconfiguration is also written into the memcached log file.

The NDB engine supports the complete set of memcache protocol commands. When a newly installed server is started with the default server role and configuration schema, you should be able to run memcapable, a memcache-server verification tool, and see all tests pass. After a configuration has been customized, however—for instance, by disabling the FLUSH_ALL command—some memcapable tests are expected to fail.

GET, SET, ADD, REPLACE, and DELETE operations.  Each of these operations is always performed according to a cache policy associated with the memcache key prefix. It may operate on a locally cached item, an item stored in the database, or both. If an operation has been disabled for the prefix, the developer should be sure to test the disabled operation, since it may fail silently, or with a misleading response code.

CAS.  CAS, in the memcache protocol, refers to a “compare and set” value, which is used as a sort of version number on a cached value, and enables some optimistic application behavior

If a container includes a CAS column, the ndb engine will generate a unique CAS ID every time it writes a data value, and store it in the CAS column.

Some memcache operations include CAS checks, such as the ASCII CAS update which has the semantics “update this value, but only if its CAS id matches the CAS id in the request”. These operations are supported by the NDB engine. The check of the stored CAS ID against the application's CAS ID is performed in an atomic operation on the NDB data node. This allows CAS checks to work correctly even when multiple memcached servers access the same key-value pair.

If CAS ID checks are in use, and additional NDB Cluster APIs other than memcached are being used to manipulate the data, then the applications using those APIs are responsible for invalidating the stored CAS IDs whenever they update data. They can do this by setting the stored CAS ID value to 0 or NULL.

The CAS ID is generated using a scheme that attempts to prevent different servers from generating overlapping IDs. This scheme can be considered a best effort, but not a guarantee, of uniqueness. The scheme constructs an initial CAS as follows:

Part of the 32-bit Cluster GCI from the primary cluster at memcached startup time is used for the high-order bits of the 64-bit CAS ID

Part of the unique cluster node id in the primary cluster used when fetching configuration is used for middle-order bits of the CAS ID

An incrementing counter in the low-order bits of the CAS ID is at least 28-bits wide.

While the NDB engine generates one sequence of CAS IDs, the default engine—used for caching values in local memcached servers—generates a different sequence. Not all combinations of CAS behavior and cache policies have been tested, so any application developer wishing to use CAS should thoroughly test whether a particular configuration behaves as desired.

FLUSH_ALL.  FLUSH_ALL is implemented as follows: First, the NDB engine iterates over all configured key-prefixes. For any prefix whose cache policy enables a database flush (flush_from_db is true), it performs a scanning delete of every row in that prefix's container table. Other prefixes are ignored. This can be a slow operation if the table is large, and some memcache clients may time out before the DELETE operation is complete. After all database deletes are complete, the FLUSH_ALL command is forwarded to the standard caching engine, which sets a flag invalidating all cached data.

INCR and DECR.  All INCR and DECR operations are pushed down to the NDB data nodes and performed atomically there. This allows multiple memcached servers to increment or decrement the same key and be guaranteed a unique value each time.

The INCR and DECR operations have clearer and more useful semantics in the binary memcache protocol than in the ASCII protocol. The binary protocol is recommended.

The memcached ASCII protocol introduces some ambiguities in the handling of INCR and DECR, and forces the NDB engine to work in dup_numbers mode, in which the value_column and the math_column must mirror each other.

dup_numbers mode is enabled for key prefixes that meet all of the following conditions:

In dup_numbers mode, the following special behavior applies:

APPEND and PREPEND.  The memcache APPEND and PREPEND operations are implemented as a single transaction which involves a read of the existing value with an exclusive lock, followed by a write of the new value. The read and write are grouped atomically into a transaction, but unlike INCR and DECR, which can run natively on the data nodes, APPEND and PREPEND are executed inside the memcached server. This means that multiple memcached servers can contend to APPEND and PREPEND the same value, and that no updates will be lost, but this contention relies on locking behavior that could cause noticably increased latency.

STATS.  A memcached server can provide many sets of statistics; use STATS KEYWORD from a login shell.

All statistics usually available from the memcached 1.6 core and the default engine are available. For instance, STATS, STATS SLABS, and STATS SETTINGS are all currently supported as described in the memcached documentation. Some special sets of statistics are available from the NDB engine, using the STATS commands described in the following list:

Whenever the NDB memcache engine is initialized, it writes a message including a timestamp and version number to its log file, as shown here:

12-Oct-2011 13:40:00 PDT NDB Memcache 5.5.15-ndb-7.2.1 started
[NDB 7.2.1; MySQL 5.5.15]

It also logs its attempt to connect to a primary cluster:

Contacting primary management server (localhost:1186) ...
·Connected to "localhost:1186" as node id 4.

Upon successfully fetching initial configuration data, the memcache engine logs a summary message describing the configuration similar to what is shown here:

Retrieved 3 key prefixes for server role "default_role"
The default behavior is that:
 GET uses NDB only
 SET uses NDB only
 DELETE uses NDB only
The 2 explicitly defined key prefixes are "mc:" () and "t:" (demo_table_tabs)
Server started with 4 threads.

The memcache engine also logs the establishment of each additional cluster connection, as shown here:

Connected to "" as node id 5.

A priming the pump... message indicates that the engine is about to prefetch a pool of transaction objects (API Connect Records). It is followed by a done ... message indicating how much time was used by prefetching. The server is not ready to respond to clients until after the prefetching is completed.

Priming the pump ...
Scheduler: using 2 connections to cluster 0
Scheduler: starting for 1 cluster; c0,f0,t1
done [0.579 sec].

Once the NDB engine has finished initializing, memcached prints a message verifying that the engine was loaded, and enumerating some of its features:

Loaded engine: NDB Memcache 5.5.15-ndb-7.2.1
Supplying the following features: compare and swap, persistent storage, LRU

If online reconfiguration is enabled, the NDB engine logs each reconfiguration, along with a summary of the new configuration, similar to what is shown here:

Received update to server role default_role
Retrieved 3 key prefixes for server role "default_role".
The default behavior is that:
 GET uses NDB only
 SET uses NDB only
 DELETE uses NDB only.
The 2 explicitly defined key prefixes are "mc:" () and "t:" (demo_table_tabs)
ONLINE RECONFIGURATION COMPLETE

On shutdown, memcached logs the shutdown sequence'a initialization and completion, and the NDB engine's scheduler logs its own shutdown as well:

Initiating shutdown
Shutting down scheduler.
Shutdown completed.

This section provides information about known issues with and design limitations of the Memcache API for NDB Cluster.

Problems with AUTO_INCREMENT.  ndbmemcache bypasses the NDB storage engine's mechanism for handling AUTO_INCREMENT columns. This means that, when you insert rows using ndbmemcache into a table having an AUTO_INCREMENT column, this column is not automatically updated. This can lead to duplicate key errors when inserts are performed later using SQL in a MySQL client application such as mysql.

To work around this issue, you can employ a sequence generator as described here.

Online schema changes not supported.  The memcached daemon does not detect online schema changes; after making such changes, you must restart the memcached daemon before the updated schema can be used by your application.

Fractional seconds.  Beginning with NDB 7.2.3, ndbmemcache supports the use of fractional seconds with the TIME, DATE, and DATETIME data types as implemented in MySQL 5.6.4 and later. ndbmemcache in previous versions of NDB Cluster does not support fractional seconds. (Bug #16593493)