Padraig O'Sullivan - Drupal

Local Development with Ariadne

Mon, 13 May 2013 00:00:00 -0700

I recently started a new development position with Blink Reaction so I needed to get somewhat serious about setting up a local Drupal development environment.

Ariadne

I was leaning towards making use of Vagrant for managing local development environments so I can easily switch between different projects or branches. I also believe Vagrant makes it easier to have as close a mirror to production locally as possible.

I discovered a very interesting project from MyPlanet Digital named vagrant-ariadne. Ariadne is a customized implementation of Vagrant and allows for easy deployment of Drupal installation profiles to a local VM. Another nice feature is that it attempts to emulate Acquia’s infrastructure. This is useful as a lot of Blink’s clients are deployed on the Acquia Cloud.

Assuming you have Vagrant, rvm and a ruby environment installed on your workstation, installing Ariadne is pretty straightforward:

j
vagrant gem install vagrant-vbguest vagrant-hostmaster vagrant-librarian
[sudo] gem install librarian rake knife-solo
git clone https://github.com/myplanetdigital/vagrant-ariadne.git
cd vagrant-ariadne
bundle install
bundle exec rake setup

Everything is now configured to boot a virtual box. Ariadne comes with a simple example that can be deployed:

j
project=example vagrant up

Once that command finishes running, the site can be viewed at http://example.dev/ (Ariadne uses vagrant-hostmaster for managing /etc/hosts entries).

A more involved cookbook is a cookbook for deploying the Web Experience Toolkit available on github also. If we wanted to deploy the master branch of this site, we could do:

bundle exec rake "init_project[https://github.com/wet-boew/ariadne-wet-boew-drupal]" project=wet-boew-drupal branch=master vagrant up

And that’s it!

Another nice feature of these deployed environments is that they are configured to allow remote debugging (relevant when setting up an IDE as mentioned later) and the actual site code is shared as an NFS mount. For example, the contents of my /etc/exports file after booting a box with Ariadne looks like:

# VAGRANT-BEGIN: 7ac1cf50-4498-4e49-bd66-edac4a9b2d7e
"/Users/posullivan/vagrant-ariadne/tmp/apt/cache" 33.33.33.10 -mapall=501:20
"/Users/posullivan/vagrant-ariadne/tmp/drush/cache" 33.33.33.10 -mapall=501:20
"/Users/posullivan/vagrant-ariadne/data/html" 33.33.33.10 -mapall=501:20
# VAGRANT-END: 7ac1cf50-4498-4e49-bd66-edac4a9b2d7e

Thus, if I navigate to the ~/vagrant-ariadne/data/html directory or import that in my IDE, I can edit the code deployed on the vagrant box.

Drupal Core from git

Another use I’ve found for ariadne is building a local environment for the latest drupal core. To accomplish this, I created a role file named roles/core.rb with the following contents:

name "core"
description "Install requirements to run Drupal core."
run_list([
  "recipe[mysql::server]",
  "recipe[mysql::client]",
  "recipe[php::module_mysql]",
  "recipe[php::module_curl]",
  "recipe[php::module_gd]",
  "recipe[php::module_apc]",
  "recipe[drush::utils]",
  "recipe[drush::make]",
  "recipe[php::write_inis]",
])
default_attributes({
  :drush => {
    :version => "5.8.0",
  },
  :mysql => {
    :server_debian_password => "root",
    :server_root_password => "root",
    :server_repl_password => "root",
    :bind_address => "127.0.0.1",
    :tunable => {
      :key_buffer => "384M",
      :table_cache => "4096",
    },
  },
})

Next, I created a new cookbook project named core and created a simple default.rb recipe for this cookbook. This recipe looks like:

branch = node['ariadne']['branch']

git "/mnt/www/html/drupal" do
  user "vagrant"
  repository "http://git.drupal.org/project/drupal.git"
  reference branch
  enable_submodules true
  action :sync
  notifies :run, "bash[Installing Drupal...]", :immediately
end

bash "Installing Drupal..." do
  user "vagrant"
  group "vagrant"
  code <<-EOH
    drush -y si \
      --root=/mnt/www/html/drupal \
      --db-url=mysqli://root:root@localhost/drupal \
      --site-name="Drupal Core Installed from Git" \
      --site-mail=vagrant+site@localhost \
      --account-mail=vagrant+admin@locahost \
      --account-name=admin \
      --account-pass=admin
  EOH
end

site = node['ariadne']['host_name'].nil? ? "#{node['ariadne']['project']}.dev" : node['ariadne']['host_name']

web_app site do
  cookbook "ariadne"
  template "drupal-site.conf.erb"
  port node['apache']['listen_ports'].to_a[0]
  server_name site
  server_aliases [ "www.#{site}" ]
  docroot "/mnt/www/html/drupal"
  notifies :reload, "service[apache2]"
end

With all of the above in place, its quite simple to create a local VM based on the latest in the 7.x branch of drupal core:

project=core branch=7.x vagrant up

The above command simply needs to have the branch name modified to deploy a different branch. Once the above command completes, a site will be available at core.dev and I can log in as the admin user using the credentials specified in my cookbook.

Private Repositories

Most repositories for client projects are stored in private repositories. Thankfully, thats not an issue with ariadne. Ariadne uses agent forwarding to forward the host machine’s ssh session into the VM, including keys and passphrases stored by ssh-agent. What this means is that your VM will have the same Git/SSH access that you enjoy on your local machine. I’ve not had a problem checking out code stored in private repositories on bitbucket for example.

IDE

For an IDE, I’ve been an Eclipse user in the past for Java projects I’ve worked on so Aptana seemed like a good fit for my needs at the moment. A few existing articles already exist on configuring Aptana for Drupal development so I’m not going to go into too much details here.

Installation is very straightforward with the binary downloaded from the site. A ruble exists for Drupal so its pretty natural to install that:

git clone git://github.com/arcaneadam/Drupal-Bundle-for-Aptana.git ~/Documents/Aptana Rubles/Drupal-Bundle-for-Aptana

Next item is to configure Aptana to adhere to the Drupal coding standards. I used an existing profile for Aptana that could be imported for this.

The final thing I needed to configure was a debug configuration. To do this, I created a new PHP web page configuration. First, a new PHP server needs to be added. In this example, lets assume I am using the example box I mentioned in the Ariadne section whose hostname is example.dev. The web server configuration dialog when configured with this hostname and appropriate directory for the site root looks like:

Once a PHP server has been added, the rest of the information to fill in for the debug configuration is pretty straightforward as shown below:

I like to select the break at first line option to make sure the debug configuration works correctly.

With this in place, any visit to example.dev will result in the breakpoint being hit.

Conclusion

I’ve still not settled on this combination for my development environment but I was definitely pretty excited upon discovering the Ariadne project. The drawbacks that I see to using Ariadne are: 1) the need to create a cookbook for each project you want to work with, 2) the project is still in beta stage so documentation is fairly lacking (fair enough for a beta project though), and 3) if you are not familiar with chef, using Ariadne may prove challenging (although it provides the perfect excuse to become familiar with chef).

PHPStorm is the IDE that seems to be pretty popular when I ask what other people are using for an editor but given there is a license fee associated with it, I didn’t want to splurge on that just yet. Aptana looks to work just fine for me and satisfies my needs nicely.

Akiban is Now Open Source

Tue, 02 Apr 2013 00:00:00 -0700

I’ve written a lot about the work I do for Akiban with Drupal in the past and many people would ask if Akiban was open source software. Well in the last few weeks we actually open sourced our database server. We also have downloads for various platforms such as Windows and OSX besides binary packages for Linux variants.

I wrote previously about how to install Drupal 7 completely on Akiban. You can still follow that post to get up and running except now there is a tiny change for our public repositories:

sudo apt-get install -y python-software-properties
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 0AA4244A
sudo add-apt-repository "deb http://software.akiban.com/apt-public/ lucid main"
sudo apt-get update
sudo apt-get install -y akiban-server postgresql-client

Some of the things included in our open source database are:

spatial indexes
full text indexes (implemented using Lucene)
REST access
nested SQL

We are also working on offering on a service offering for our database server so there will be no need to manage an installation yourself. If you are interested in trying our service in its current beta form, please let me know in the comments or hit me up on twitter and I’d be happy to hook you up or just visit our website. We also have a public mailing list for the Akiban project if you try anything out and have any questions.

Akiban as a MySQL Replica with Drupal 7

Fri, 11 Jan 2013 00:00:00 -0800

I previously wrote about how to install Drupal 7 completely on Akiban. However, this is not how our current customers are using us. The vast majority of all Drupal installations currently run on MySQL. What we at Akiban are currently aiming to do is to be deployed as a regular MySQL slave and if there are any queries that are problematic for MySQL, we work with customers to make sure those queries get executed by Akiban (and with a significant performance improvement).

In this post, I wanted to cover how to setup Akiban as a MySQL slave and how a query is typically re-directed to an Akiban server from Drupal. This article is specific to Drupal 7.

First, I setup a regular Drupal install on Ubuntu 12.04 with MySQL 5.5.28. This is going to serve as the master server. To configure replication in MySQL is pretty straightforward. The following needs to be placed in your my.cnf file and MySQL needs to be re-started:

log-bin=mysql-bin
server-id=11

A user needs to be created for replication:

CREATE USER 'repl'@'%' IDENTIFIED BY 'password';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
FLUSH PRIVILEGES;

Next steps are to take a consistent snapshot of your Drupal schema with mysqldump and capture the output of SHOW MASTER STATUS to get the appropriate binlog co-ordinates.

Next, we need to setup an Akiban MySQL slave. We will use an entirely separate instance for this purpose. First, the software to install on this slave is:

sudo apt-get install -y mysql-client mysql-server
sudo apt-get install -y python-software-properties
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 0AA4244A
sudo add-apt-repository "deb http://software.akiban.com/apt-developer/lucid main"
sudo apt-get update
sudo apt-get install -y akiban-server akiban-adapter-mysql postgresql-client
echo "INSTALL PLUGIN akibandb SONAME 'libakibandb_engine.so'" | mysql -u
root

Issuing the SHOW PLUGINS command on this slave will now show the AkibanDB storage engine. The next step is to now import the mysqldump file taken from the master and configure replication. On the slave server, you need to make sure server-id is set in the my.cnf file. Then to enable replication, a CHANGE MASTER command needs to be issued. An example of what that command might look like is:

CHANGE MASTER TO
  MASTER_HOST = 'ec2-23-20-112-161.compute-1.amazonaws.com',
  MASTER_USER = 'repl',
  MASTER_PASSWORD = 'password',
  MASTER_LOG_FILE = 'mysql-bin.000001',
  MASTER_LOG_POS = 403

Finally, issuing START SLAVE starts up replication. The observant among you will notice all tables are still InnoDB on the slave. We have done nothing to convert any tables to Akiban yet. Before we get to that I want to configure Drupal running on the master server to know about the Akiban slave so it can send queries to it. First, we need to install the Akiban database module in Drupal (the akiban directory should be copied to whatever the appropriate location for your Drupal install is) and the PHP client drivers for PostgreSQL:

sudo apt-get install -y git php5-pgsql
git clone http://git.drupal.org/sandbox/posulliv/1835778.git akiban
cd akiban
git checkout 7.x
cd ../
sudo cp -R akiban /var/www/drupal/includes/database/.

Now, the settings.php file needs to be updated to know about this Akiban server:

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
    ),
    'slave' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => 'ec2-23-22-113-161.compute-1.amazonaws.com',
      'port' => '15432',
      'driver' => 'akiban',
      'prefix' => '',
    ),
  ),
);

I would suggest enabling query logging on the Akiban server so you can see read queries being sent to the slave. Query logging can be enabled by modifying the /etc/akiban/config/server.properties file to have these entries:

akserver.querylog.enabled=true
akserver.querylog.filename=/var/log/akiban/queries.log
akserver.querylog.exec_time_threshold=0

All queries issued against Akiban will now be logged to the /var/log/akiban/queries.log file since we set the query execution time threshold to 0. Akiban needs to re-started for this to take effect.

By default, very few queries from Drupal core are sent to a slave database. The search module is probably the best module to test with to see queries being sent to Akiban. The search module can be accessed from your Drupal site by going to http://your.ip.address/drupal/?q=search

First, we need to convert those tables to Akiban, otherwise any search will now fail since no tables have been converted to Akiban yet. To convert these tables to Akiban, we simply issue the following in MySQL:

STOP SLAVE;
ALTER TABLE search_total ENGINE=AkibanDB;
ALTER TABLE search_index ENGINE=AkibanDB;
ALTER TABLE node ENGINE=AkibanDB;
ALTER TABLE search_index ADD CONSTRAINT `__akiban_fk_00` FOREIGN KEY (sid) REFERENCES node (nid);
ANALYZE TABLE node;
ANALYZE TABLE search_index;
ANALYZE TABLE search_total;
START SLAVE;

The relevant tables are now converted to Akiban. Now, try searching content for a keyword. If everything is working correctly, queries should start appearing in the query log on the Akiban server when issuing content searches.

This is obviously a pretty simple example but now its pretty trivial to send more queries to Akiban. Just change the database target, convert the appropriate tables to Akiban on the slave, and away you go!

If there is anything you would like more information on, please let me know in the comments or hit me up on twitter and I’d be more than happy to dig in. We also have a public mailing list for the Akiban project and I’d encourage anyone who’s interested to subscribe to that list and let us know how we’re doing! Finally, I’ll be presenting on this topic at drupalcamp MA on January 19th and I am also delivering a joint webinar with Acquia in February on this topic.

Testing an Alternate Field SQL Storage Module

Tue, 08 Jan 2013 00:00:00 -0800

After my post yesterday testing the field storage layer, a commentator pointed out an alternate SQL storage module that does not create a revision table for each field. Naturally, I had to try this out to see how what kind of performance was possible with this approach.

The average throughput numbers I observed using this module are shown in the table below.

Environment	Average Throughput
Default MySQL	2892 nodes / minute
Default PostgreSQL	2313 nodes / minute
Tuned MySQL	4730 nodes / minute
Tuned PostgreSQL	2464 nodes / minute

The image below shows the results graphically for different environments I tested. The Y axis is throughput (node per minute) with the X axis specifying the CSV file (corresponding to a MLB year) being imported.

That’s a pretty big improvement over the numbers I got in my original test. We still are not approaching the 8000 nodes per minute that is possible with a tuned MySQL instance and MongoDB for field storage but at about 5000 nodes per minute, we are getting somewhat close. It does beg the question of whether the performance benefits of MongoDB for field storage are worth it when we can get somewhat close using this module and a site’s original database system?

I would be interested in suggestions for read benchmarks from the community for different field storage backends so I can attempt to gain more insight into this question for myself.

Field Storage Tests with Drupal 7

Mon, 07 Jan 2013 00:00:00 -0800

I had some spare time this weekend and decided to do some tests with the field storage layer. I really just wanted to re-produce the results Moshe Weitzman published a while back. I also wanted to see what the best results I could get were.

Environment Details

The software and versions used for testing were:

EC2 EBS backed Large instance (8GB of memory) in the US-EAST availability zone
Ubuntu 12.04 (ami-fd20ad94 as listed in official ubuntu AMI’s)
MySQL 5.5.28
PostgreSQL 9.2
MongoDB 2.0.4
Drupal 7.17
Drush 5.1
Migrate 2.5

I ran tests against both MySQL and PostgreSQL with default settings for both but I also ran tests where I modified the configuration of both systems to be optimized for writes.

The configuration options I specified for MySQL when tuning it were:

innodb_flush_log_at_trx_commit=0
innodb_doublewrite=0
log-bin=0
innodb_support_xa=0
innodb_buffer_pool_size=6G
innodb_log_file_size=512M

The configuration options I specified for PostgreSQL when tuning it were:

fsync = off
synchronous_commit = off
wal_writer_delay = 10000ms
wal_buffers = 16MB
checkpoint_segments = 64
shared_buffers = 6GB

Dataset

The dataset used for the tests comes from the migrate_example_baseball module that comes as part of the migrate module. This dataset contains a box score from every Major League Baseball game from the year 2000 to the year 2009. Each year’s data is contained in CSV file. Different components of the box score are saved in fields hence stressing field storage a lot.

Results

Average throughput numbers for the various configurations I tested are shown in the table below.

Environment	Average Throughput
Default MySQL	1932 nodes / minute
Default PostgreSQL	1649 nodes / minute
Tuned MySQL	3024 nodes / minute
Tuned PostgreSQL	1772 nodes / minute
Default MySQL with MongoDB	4609 nodes / minute
Default PostgreSQL with MongoDB	4810 nodes / minute
Tuned MySQL with MongoDB	7671 nodes / minute
Tuned PostgreSQL with MongoDB	5911 nodes / minute

Conclusion

Its pretty obvious from glancing at the results above that using MongoDB for field storage results in the best throughput. Tuned MySQL using MongoDB for field storage gave me the best results. This is consistent with what Moshe reported in his original article as well.

What was very interesting to me was the PostgreSQL numbers. The overhead of having a table per field with the default SQL field storage seems to be very high with PostgreSQL. Its interesting to see how much better an optimized PostgreSQL does when using MongoDB for field storage.

After performing these tests, one experiment I really want to try now is to create a field storage module for PostgreSQL that uses the JSON data type included in the 9.2 release. Hopefully, I will get some spare time in the coming week or two to work on that.

Making Drupal more RESTful with Akiban

Mon, 17 Dec 2012 00:00:00 -0800

Last week, I published an article on how to install Drupal 7 with Akiban as the backend database. Today, I wanted to briefly show off our REST API using the schema that is created with a standard install of Drupal 7 core.

First, I installed the devel module and generated some data since a bare bones install with no data would not be much fun. This server is running on a publically available EC2 instance too so if you are interested in trying these examples out yourself at home, feel free to do so! I’ll leave the EC2 instance up and running for the remainder of 2012 but if anyone wants to try the examples out and the instance seems unavailable, please let me know and I’ll fire it up again for you.

For the first few examples, I’m going to use curl since its available on nearly every system (including OSX). Lets first get the version of the Akiban we are going to be interacting with:

$ curl -X GET -H "Content-Type: application/json" http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/version
[
{"server_name":"Akiban Server","server_version":"1.4.4.2451"}
]
$

Lets continue this trend of a few simple examples to get started. I want to know the list of schemas on this server I am interacting with:

$ curl -X GET -H "Content-Type: application/json" http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/information_schema.schemata
[
{"schema_name":"drupal","schema_owner":null,"default_character_set_name":null,"default_collation_name":null},
{"schema_name":"information_schema","schema_owner":null,"default_character_set_name":null,"default_collation_name":null},
{"schema_name":"sqlj","schema_owner":null,"default_character_set_name":null,"default_collation_name":null},
{"schema_name":"sys","schema_owner":null,"default_character_set_name":null,"default_collation_name":null},
{"schema_name":"test","schema_owner":null,"default_character_set_name":null,"default_collation_name":null}
]
$

Lets try a Drupal specific example next. Our REST API allows you to retrieve an entire table group in 1 request. So let’s say I wanted to get all information for a certain user (I pretty-printed the JSON in the output below so if you run this you will need to format the output):

$ curl -X GET -H "Content-Type: application/json" http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/drupal.users/1
[
    {
        "uid": 1,
        "name": "posulliv",
        "pass": "$S$DPV31LZyFWJmJ.Fcj6IRyjb/RFMyQQtE87gsad7cavgnH3fw0GHA",
        "mail": "posullivan@akiban.com",
        "theme": "",
        "signature": "",
        "signature_format": null,
        "created": 1355345142,
        "access": 1355762571,
        "login": 1355345211,
        "status": 1,
        "timezone": "America/New_York",
        "language": "",
        "picture": 0,
        "init": "posullivan@akiban.com",
        "data": "YjowOw==",
        "drupal.authmap": [],
        "drupal.sessions": [
            {
                "uid": 1,
                "sid": "jq57PowPwDK1CuKBpC56oqt_PsbwWNF4av97BuQqr6I",
                "ssid": "",
                "hostname": "75.147.9.1",
                "timestamp": 1355762574,
                "cache": 0,
                "session": "YmF0Y2hlc3xhOjE6e2k6MTtiOjE7fWF1dGhvcml6ZV9maWxldHJhbnNmZXJfaW5mb3xhOjE6e3M6MzoiZnRwIjthOjU6e3M6NToidGl0bGUiO3M6MzoiRlRQIjtzOjU6ImNsYXNzIjtzOjE1OiJGaWxlVHJhbnNmZXJGVFAiO3M6NDoiZmlsZSI7czo3OiJmdHAuaW5jIjtzOjk6ImZpbGUgcGF0aCI7czoyMToiaW5jbHVkZXMvZmlsZXRyYW5zZmVyIjtzOjY6IndlaWdodCI7aTowO319YXV0aG9yaXplX29wZXJhdGlvbnxhOjQ6e3M6ODoiY2FsbGJhY2siO3M6Mjg6InVwZGF0ZV9hdXRob3JpemVfcnVuX2luc3RhbGwiO3M6NDoiZmlsZSI7czozNToibW9kdWxlcy91cGRhdGUvdXBkYXRlLmF1dGhvcml6ZS5pbmMiO3M6OToiYXJndW1lbnRzIjthOjM6e3M6NzoicHJvamVjdCI7czo1OiJkZXZlbCI7czoxMjoidXBkYXRlcl9uYW1lIjtzOjEzOiJNb2R1bGVVcGRhdGVyIjtzOjk6ImxvY2FsX3VybCI7czozNzoiL3RtcC91cGRhdGUtZXh0cmFjdGlvbi1kOWU4MTUzOS9kZXZlbCI7fXM6MTA6InBhZ2VfdGl0bGUiO3M6MTQ6IlVwZGF0ZSBtYW5hZ2VyIjt9bWVzc2FnZXN8YToxOntzOjU6ImVycm9yIjthOjI6e2k6MDtzOjI3NToiPGVtIGNsYXNzPSJwbGFjZWhvbGRlciI+V2FybmluZzwvZW0+OiBhcnJheV9rZXlfZXhpc3RzKCkgZXhwZWN0cyBwYXJhbWV0ZXIgMiB0byBiZSBhcnJheSwgbnVsbCBnaXZlbiBpbiA8ZW0gY2xhc3M9InBsYWNlaG9sZGVyIj50aGVtZV9pbWFnZV9mb3JtYXR0ZXIoKTwvZW0+IChsaW5lIDxlbSBjbGFzcz0icGxhY2Vob2xkZXIiPjYwNTwvZW0+IG9mIDxlbSBjbGFzcz0icGxhY2Vob2xkZXIiPi92YXIvd3d3L2RydXBhbC9tb2R1bGVzL2ltYWdlL2ltYWdlLmZpZWxkLmluYzwvZW0+KS4iO2k6MTtzOjI3NToiPGVtIGNsYXNzPSJwbGFjZWhvbGRlciI+V2FybmluZzwvZW0+OiBhcnJheV9rZXlfZXhpc3RzKCkgZXhwZWN0cyBwYXJhbWV0ZXIgMiB0byBiZSBhcnJheSwgbnVsbCBnaXZlbiBpbiA8ZW0gY2xhc3M9InBsYWNlaG9sZGVyIj50aGVtZV9pbWFnZV9mb3JtYXR0ZXIoKTwvZW0+IChsaW5lIDxlbSBjbGFzcz0icGxhY2Vob2xkZXIiPjYwNTwvZW0+IG9mIDxlbSBjbGFzcz0icGxhY2Vob2xkZXIiPi92YXIvd3d3L2RydXBhbC9tb2R1bGVzL2ltYWdlL2ltYWdlLmZpZWxkLmluYzwvZW0+KS4iO319"
            }
        ],
        "drupal.shortcut_set_users": [],
        "drupal.users_roles": [
            {
                "uid": 1,
                "rid": 3
            }
        ],
        "drupal.watchdog": [
            {
                "wid": 160662,
                "uid": 1,
                "type": "php",
                "message": "JXR5cGU6ICFtZXNzYWdlIGluICVmdW5jdGlvbiAobGluZSAlbGluZSBvZiAlZmlsZSku",
                "variables": "YTo2OntzOjU6IiV0eXBlIjtzOjc6Ildhcm5pbmciO3M6ODoiIW1lc3NhZ2UiO3M6NjI6ImFycmF5X2tleV9leGlzdHMoKSBleHBlY3RzIHBhcmFtZXRlciAyIHRvIGJlIGFycmF5LCBudWxsIGdpdmVuIjtzOjk6IiVmdW5jdGlvbiI7czoyMzoidGhlbWVfaW1hZ2VfZm9ybWF0dGVyKCkiO3M6NToiJWZpbGUiO3M6NDU6Ii92YXIvd3d3L2RydXBhbC9tb2R1bGVzL2ltYWdlL2ltYWdlLmZpZWxkLmluYyI7czo1OiIlbGluZSI7aTo2MDU7czoxNDoic2V2ZXJpdHlfbGV2ZWwiO2k6NDt9",
                "severity": 4,
                "link": "0",
                "location": "aHR0cDovL2VjMi01MC0xOS0yOC0yNy5jb21wdXRlLTEuYW1hem9uYXdzLmNvbS9kcnVwYWwv",
                "referer": "aHR0cDovL2VjMi01MC0xOS0yOC0yNy5jb21wdXRlLTEuYW1hem9uYXdzLmNvbS9kcnVwYWwv",
                "hostname": "24.61.45.238",
                "timestamp": 1355406786
            }
        ]
    }
]
$

We also support multi-get so you can retrieve a number of table groups in a single REST API call. For example, lets say I want to get information on 2 users:

$ curl -X GET -H "Content-Type: application/json" "http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/drupal.users/11467;10503"
[
    {
        "uid": 11467,
        "name": "clibriprofr",
        "pass": "",
        "mail": "clibriprofr@default",
        "theme": "",
        "signature": "",
        "signature_format": null,
        "created": 1355360324,
        "access": 0,
        "login": 0,
        "status": 1,
        "timezone": null,
        "language": "und",
        "picture": 11463,
        "init": "",
        "data": null,
        "drupal.authmap": [],
        "drupal.sessions": [],
        "drupal.shortcut_set_users": [],
        "drupal.users_roles": [],
        "drupal.watchdog": []
    },
    {
        "uid": 10503,
        "name": "uuslosuwr",
        "pass": "",
        "mail": "uuslosuwr@default",
        "theme": "",
        "signature": "",
        "signature_format": null,
        "created": 1355360324,
        "access": 0,
        "login": 0,
        "status": 1,
        "timezone": null,
        "language": "und",
        "picture": 10499,
        "init": "",
        "data": null,
        "drupal.authmap": [],
        "drupal.sessions": [],
        "drupal.shortcut_set_users": [],
        "drupal.users_roles": [],
        "drupal.watchdog": []
    }
]
$

Our REST API also supports aribtrary SQL queries being executed and results being returned as JSON. Lets try a simple example first:

$ curl -X GET -H "Content-Type: application/json" "http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/query?q=select%20count(*)%20from%20drupal.comment"
[
{"_SQL_COL_1":252462}
]
$

Another example of executing arbitrary SQL queries through our REST API with a more complex query follows. The query we will use for this example is:

SELECT c.* 
FROM   drupal.comment c 
       INNER JOIN drupal.node n 
               ON n.nid = c.nid 
WHERE  ( c.status = 1 ) 
       AND ( n.status = 1 ) 
ORDER  BY c.created DESC, 
          c.cid DESC 
LIMIT  10 offset 0

Running this query through our REST API and the result (again, nicely formatted) looks like:

curl -X GET -H "Content-Type: application/json" "http://ec2-50-19-28-27.compute-1.amazonaws.com:8091/api/query?q=SELECT%20c.*%20FROM%20drupal.comment%20c%20INNER%20JOIN%20drupal.node%20n%20ON%20n.nid%20=%20c.nid%20WHERE%20(c.status%20=%201)%20AND%20(n.status%20=%201)%20ORDER%20BY%20c.created%20DESC,%20c.cid%20DESC%20LIMIT%2010%20OFFSET%200"
[
    {
        "cid": 304562,
        "pid": 0,
        "nid": 93450,
        "uid": 1,
        "subject": "this is a test comment",
        "hostname": "75.147.9.1",
        "created": 1355418376,
        "changed": 1355418376,
        "status": 1,
        "thread": "01/",
        "name": "posulliv",
        "mail": "",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304561,
        "pid": 304558,
        "nid": 93451,
        "uid": 3636,
        "subject": "Defui Enim Gemino Luctus Occuro Paulatim",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "01.00/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304560,
        "pid": 0,
        "nid": 93451,
        "uid": 3633,
        "subject": "Abdo Ea Sudo Veniam Vulputate",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "03/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304559,
        "pid": 0,
        "nid": 93451,
        "uid": 3651,
        "subject": "Defui Euismod Letalis Nisl Utinam Vicis",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "02/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304558,
        "pid": 0,
        "nid": 93451,
        "uid": 3657,
        "subject": "Similis Te",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "01/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304557,
        "pid": 0,
        "nid": 93448,
        "uid": 3630,
        "subject": "Loquor Modo Ut",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "02/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304556,
        "pid": 0,
        "nid": 93448,
        "uid": 3648,
        "subject": "Abico Conventio Elit Quis",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "01/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304555,
        "pid": 0,
        "nid": 93447,
        "uid": 3646,
        "subject": "Dolor Immitto Metuo Veniam",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "04/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304554,
        "pid": 304553,
        "nid": 93447,
        "uid": 3633,
        "subject": "Defui Et Pertineo Premo Usitas",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "01.00.00/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    },
    {
        "cid": 304553,
        "pid": 304550,
        "nid": 93447,
        "uid": 3655,
        "subject": "Amet Gravis Inhibeo Roto Torqueo",
        "hostname": "127.0.0.1",
        "created": 1355369527,
        "changed": 1355369527,
        "status": 1,
        "thread": "01.00/",
        "name": "devel generate",
        "mail": "devel_generate@example.com",
        "homepage": "",
        "language": "und"
    }
]
$

Finally, I’d like to mention we have a client for node.js that can be used with our REST interface. To get information on the schemas in this server and the grouping in the drupal schema, some code using this client would look as follows:

#!/usr/bin/env coffee

ak = require('./lib/akiban_rest.js')
_  = require('underscore')

log = (msg) ->
  () ->
    console.log("========")
    console.log(msg)
    console.log("--------")
    unless arguments[0].error
      _(arguments[0].body).forEach (x) ->
        console.log(x)
    console.log(arguments) if arguments[0].error
    console.log("--------")

x = new ak.AkibanClient()

x.version(log('the server version is'))
x.schemata(log('and these are all the schemata'))
x.groups('drupal', log('all groups in the drupal schema'))

The above can be run with the coffee command like so: coffee drupal.coffee.

To retrieve a certain node with this client, the code would look like:

#!/usr/bin/env coffee

ak = require('./lib/akiban_rest.js')
_  = require('underscore')

log = (msg) ->
  () ->
    console.log("========")
    console.log(msg)
    console.log("--------")
    unless arguments[0].error
      _(arguments[0].body).forEach (x) ->
        console.log(x)
    console.log(arguments) if arguments[0].error
    console.log("--------")

x = new ak.AkibanClient()
x.get 'drupal', 'node', 2054, (res) -> log('retrieving nid 2054')(res)

Running the above example results in output like:

$ coffee drupal.coffee 
========
retrieving nid 2054
--------
{ nid: 2054,
  vid: 2054,
  type: 'page',
  language: 'und',
  title: 'Eros Iriure Pertineo Refoveo Roto Utrum',
  uid: 3661,
  status: 1,
  created: 1355369527,
  changed: 1355369527,
  comment: 0,
  promote: 1,
  sticky: 0,
  tnid: 0,
  translate: 0,
  'drupal.comment': [],
  'drupal.history': [],
  'drupal.node_access': [],
  'drupal.node_comment_statistics': 
   [ { nid: 2054,
       cid: 0,
       last_comment_timestamp: 1355369527,
       last_comment_name: null,
       last_comment_uid: 3656,
       comment_count: 0 } ],
  'drupal.node_revision': 
   [ { nid: 2054,
       vid: 2056,
       uid: 1,
       title: 'Ad Si Suscipere',
       log: '',
       timestamp: 1355369527,
       status: 1,
       comment: 0,
       promote: 1,
       sticky: 0 } ],
  'drupal.search_node_links': [] }
--------

Thats about it for this post showing off our REST access. As usual, comments are very much welcome and feel free to ping me on twitter if you would like to learn more about Akiban.

Installing Drupal 7 with Akiban

Fri, 14 Dec 2012 00:00:00 -0800

Dries recently published a post highlighting some work we’ve done with a particular customer in the Acquia cloud. What I wanted to cover in this post was to how to perform an installation of Akiban and get a Drupal 7 site up and running on Akiban. This post only covers a fresh installation; later posts will cover how to do migration and augmenting an existing site instead of running it entirely on Akbian.

This post is specific to Ubuntu but Akiban runs on CentOS too (as well as OSX and Windows which we have installers for). If people would like to see information specific to those platforms, please let me know in the comments.

First things first, lets install Akiban!

sudo apt-get install -y python-software-properties
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 0AA4244A
sudo add-apt-repository "deb http://software.akiban.com/apt-developer/ lucid main"
sudo apt-get update
sudo apt-get install -y akiban-server postgresql-client

The above will automatically start the Akiban server process and half of your available memory will be allocated for the JVM heap by default. If interested in modifying any configuration options, please see our documention on how to do this.

Next, we’ll download Drupal 7 and install Apache along with the needed PHP database drivers for Akiban.

wget http://ftp.drupal.org/files/projects/drupal-7.17.tar.gz
tar zxvf drupal-7.17.tar.gz
sudo apt-get install -y apache2 php5-pgsql php5-gd libapache2-mod-php5 php-apc
sudo mkdir /var/www/drupal
sudo mv drupal-7.17/* drupal-7.17/.htaccess /var/www/drupal
sudo cp /var/www/drupal/sites/default/default.settings.php /var/www/drupal/sites/default/settings.php
sudo chown www-data:www-data /var/www/drupal/sites/default/settings.php
sudo mkdir /var/www/drupal/sites/default/files
sudo chown www-data:www-data /var/www/drupal/sites/default/files/
sudo service apache2 restart

The final piece of software we need is the Akiban database module for Drupal. Right now, this module is a sandbox project on drupal.org so the only way to download it is to check it out with git:

sudo apt-get install -y git
git clone http://git.drupal.org/sandbox/posulliv/1835778.git akiban
cd akiban
git checkout 7.x
cd ../
sudo cp -R akiban /var/www/drupal/includes/database/.
sudo chown -R www-data:www-data /var/www/drupal/includes/database/akiban

Notice we had to switch to the 7.x branch. The master branch in this repository is for running the module with Drupal 8.

The last thing which needs to be done is apply a tiny patch to Drupal core. This patch only avoids the creation of 2 indexes in the menu module. These index defitions are not compatible with Akiban with our current release. Its likely this will be resolved in a future Akiban release and so the need for this patch will be removed:

sudo cp akiban/core.patch /var/www/drupal
cd /var/www/drupal
sudo patch -p1 < core.patch
cd

Drupal 7 can now be installed as you normally would. Just make sure to select Akiban as the database during installation!

After installation completes successfully we want to group the tables and gather statistics for out cost-based optimizer. 2 SQL scripts are provided to achieve this. They can be run using psql as so:

psql -h localhost -p 15432 drupal -f akiban/grouping.sql
psql -h localhost -p 15432 drupal -f akiban/gather_stats.sql

The commands above assume drupal is the name of schema in which Drupal was installed. That should obviously be changed to the name of the schema you specified during installation.

Thats it! You now have a bare Drupal 7 site running on the Akiban database! I have some plans for more posts in the coming weeks. In particular, some things I plan on covering are how to migrate a Drupal site running on MySQL to Akiban and how to use Akiban as a query accelerator for a Drupal site similar to the use case in the post Dries wrote. I’ll also show what is possible with the REST access that we enable straight to our database (hint: its on port 8091).

Digging into Drupal's Schema

Thu, 02 Aug 2012 00:00:00 -0700

I’m relatively new to Drupal internals and most of the work I do is on the database side. While searching for information on Drupal’s schema, I found very little. During my research, I put together an ER diagram of the schema installed by Drupal 7 (D8 is very similar with only 3 extra tables at time of writing) and decided to share my work. Note that the relationships I discuss here are based on the foreign key documentation that exists in core and my understanding of what I believe other relationships could be. Corrections and comments are very much welcome.

Overview

I’ll start off by showing my complete ER diagram below. You will see I grouped tables I found to be related in colored boxes. The image below is just meant to give a general overview of the schema. I will be diving into different parts of the schema in this post. I created this diagram using MySQL Workbench and the model can be downloaded from here if someone wishes to open this up in Workbench. This gist also shows the ALTER TABLE SQL statements that would need to be issued to actually create these foreign keys in MySQL. I would not recommend doing this right now with Drupal as many things would break.

Without delving into the relationships and details of this diagram, lets first cover some basic details. A stock install of Drupal 7 results in 73 tables being created. 10 of those tables are used for caching purposes:

Caching Table	Description
cache	caches items not separated out into their own cache tables
cache_block	the block modules can cache already built blocks here
cache_bootstrap	data required during the bootstrap process can be cached in this table
cache_field	stores cached field values
cache_filter	caches already filtered pieces of text
cache_form	caches recently built forms and their storage data
cache_image	caches information about image manipulations that are in progress
cache_menu	caches router information as well as generated link trees
cache_page	caches compressed pages served to anonymous users
cache_path	caches path aliases

11 tables are created which do not relate to any other tables:

Table Name	Description
actions	stores action information
batch	stores details about batches (processes that run in multiple HTTP requests)
blocked_ips	stores a list of blocked IP addresses
flood	controls the threshold of events, such as the number of contact attempts
queue	stores items in queues
rdf_mapping	stores custom RDF mappings for user-defined content types
semaphore	stores semaphores, locks, and flags
sequences	stores IDs
system	contains a list of all modules, themes, and theme engines that are or have been installed
url_alias	contains a list of URL aliases for Drupal paths
variable	stores variable/value pairs created by Drupal core or any other module or theme

The 21 tables listed above are self-explanatory and I’m not going to discuss them any further in this post. They also are independent in that these tables have no relationships with other tables.

Field Related Tables

There are 8 tables installed with core related to fields and field storage:

Table Name	Description
field_data_body	stores details about the body field of an entity
field_revision_body	stores information about revisions to body fields
field_data_comment_body	stores information about comments associated with an entity
field_revision_comment_body	stores information about revisions to comments
field_data_field_image	stores information about images associated with an entity
field_revision_field_image	stores information about revisions to images
field_data_field_tags	stores information about tags associated with an entity
field_revision_field_tags	stores information about revisions to taxonomy terms/tags associated with an entity

While I was initially tempted to have these tables related to node, that would not really be correct since these tables are related to an entity. In D7, entities can be other objects besides nodes, such as users or comments. The entity_type column in these tables reflects that reality. These tables can be stored in other storage systems such as MongoDB due to the field storage API introduced in Drupal 7.

There are 2 other tables related to fields: field_config and field_config_instance. These tables store field configuration information. I believe a row in field_config_instance cannot (well at least should not) exist without the correspondong field_id in the field_config table. Hence, the one-to-many relationship from field_config to field_config_instance is an identifying relationship.

Small Groups of Tables

There are a number of groups you will notice in the full ER diagram that are made up of 2 to 3 tables. Zooming in on 4 of those groups, we can see those tables more clearly:

One thing you will notice is that some relationships are shown with a solid line whereas others use a dotted line. MySQL Workbench represents identifying relationships with a solid line and non-identifying relationships with a dotted line. If you are unfamiliar with those terms, the standard defintions are:

identifying relationship - the foreign key attribute is part of the child’s primary key attribute.
non-identifying relationship - the primary key attributes of the parent must not become primary key attributes of the child.

This stack overflow answer from Bill Karwin contains a good discussion on these topics.

Now lets discuss those groups in more detail.

Registry Group

I grouped the registry and registry_file tables together. These tables are used for implementing the code registry in Drupal. A one-to-many relationship exists from registry_file to registry and this relationship is an identifying relationship. A filename should not appear in the registry table that is not present in the registry_file table.

Image Group

I grouped the image_styles and image_effects tables together. These tables store configuration options for image styles and effects. A one-to-many relationship exists from image_styles to image_effects and this relationship is a non-identifying relationship.

date_format Group

There are three tables about date formats in Drupal. date_format_type is a lookup table that stores configured date format types. After a stock install of Drupal 7, three date format types exist:

long
medium
short

A one-to-many relationship exists from this lookup table to both date_formats and date_format_locale.

In practice, this would be problematic. For example, a new date format can be created by an adminstrator. In D7, this results in the system_date_format_save function being called. This function will insert a row in the date_formats table that will not have a corresponding type (type will be listed as custom).

You will also notice the locked column is redundant in the date_formats table. I submitted a patch to change this.

File Group

I grouped the file_managed and file_usage tables into 1 group. These tables store information about uploaded files and information for tracking where a file is used.

I believe a 1-to-1 relationship exists from file_managed to file_usage and that this is an identifying relationship.

User Related Tables

There are quite a few tables that store user related information. Below is a figure where I zoom in on those tables.

As you can see, the tables directly associated with users are watchdog, sessions, and authmap. These tables are in a one-to-many relationship from users. The functionality these tables provide is:

Table Name	Description
authmap	stores distributed authentication mapping
sessions	stores information about a users session
watchdog	contains logs of all system events

There are then two tables that are in a many-to-many relationship with users that link this table with other groups. One of these is the users_roles table. This table links users with role. The role table is then in a one-to-many relationship with the role_permission table. The other many-to-many table is shortcut_set_users. This table links users with shortcut_set.

The tables for the menu system are not really related to users but I placed the group close by since the menu_links table maintains a one-to-many relationship with the shortcut_set table. While the tables for the menu system do not appear to be related, I do believe a relationship exists there. In particular, I think that the menu_link table has relationships to both the menu_router and menu_custom tables. The router_path column in menu_links could reference the router column in menu_router and the menu_name column in menu_links could reference the menu_name in the menu_custom table. Right now however, after a stock install of D7, a row with a menu name that is not present in menu_custom will be created in menu_links.

The menu system tables and a description of what they do is below.

Table Name	Description
menu_custom	holds definitions for top-level custom menus
menu_links	contains the individual links within a menu
menu_router	maps paths to various callbacks

Node Related Tables

Node is one of the most central concepts in Drupal so as you can imagine, many tables are related to that concept. First off, a high level overview of the tables related to the node table are shown below.

Tables that are directly related to node are node_revision, node_access, and node_type. The node_type table is in many-to-many relationship with node and block_node_type. node_revision is in a many-to-one relationship with node as is node_access. The node_access table has only 1 row upon initial installation and references a non-existent node. An issue has been created to address this.

The tables directly related to node and a description of what they do is below.

Table Name	Description
node_access	identifies which realm/grant pairs a user must possess in order to view, update, or delete specific nodes
node_revision	stores information about each saved version of a node
node_type	stores information about all defined node types

Taxonomy Tables

Four tables in the stock schema are related to taxonomy. These tables are shown in the figure below.

First of all, the taxonomy_index table is in a many-to-many relationship with the node and taxonomy_term_data tables. The taxonomy_vocabulary table has a one-to-many relationship with the taxonomy_term_data table. The taxonomy_term_data table in turn has 2 1-to-many relationships with the taxonomy_term_hierarchy table.

A description of the taxonomy tables is given below.

Table Name	Description
taxonomy_index	maintains de-normalized information about node/term relationships
taxonomy_term_data	stores term information
taxonomy_term_hierarchy	stores the hierarchical relationship between terms
taxonomy_vocabulary	stores vocabulary information

Block Tables

The main table in this group is block. It has three directly related tables in one-to-many relationships: block_node_type, block_role, and block_custom.

A description of the blocks tables is given below.

Table Name	Description
blocks	stores block settings
block_custom	stores the contents of custom-made blocks
block_node_type	stores information that sets up display criteria for blocks based on content type
block_role	stores access permissions for blocks based on user roles

Search Tables

The relationships for the search tables I am a little unsure of. I believe they are as shown in the figure below.

The relationship I’m most unsure of here are between search_total and search_index. I don’t think the one-to-many relationship I have in place from search_total to search_index is correct.

A description of the search tables is given below.

Table Name	Description
search_dataset	stores items that will be searched
search_index	stores the search index and associates words, items, and scores
search_node_links	stores items that link to other nodes
search_total	stores search totals for words

Tables That Relate Nodes to Users

There are three tables in many-to-many relationships between node and users:

Table Name	Description
comment	stores comments and associated data
history	stores a record of which users have read which nodes
node_comment_statistics	maintains statistics of nodes and comments posts to show new and updated flags

The comment table could be in its own group. I decided against doing that in this ER diagram since I felt like it would have been a table by itself. Logically, I think of it as either being in the users or node group.

node_comment_statistics does also maintain a relationship with comment. This is a non-identifying relationship since a node can exist without any comments.

Conclusion

During this work, I noticed that the column definitions for many foreign key relationships are in-correct which would result in MySQL not allowing these constraints to actually be created. I created an issue and patch for this but it turns out Liam Morland is working on using foreign keys in core and also came across this around the same time as me.

Other issues I encountered have also been logged by Liam:

the node_access table references a non-existent node (relevant issue)
a set name exists in shortcut_set that does not exist in menu_links (relevant issue)

I would vote for foreign keys being used in Drupal core for a number of reasons, not least of which foreign keys aid a newcomer when trying to understand the schema installed by Drupal.

As I mentioned at the beginning of this post, any comments or corrections are very much welcome. I hope this information can prove useful to someone else besides me!

Configuring Drupal 7.x With PostgreSQL Replication

Sun, 08 Jul 2012 00:00:00 -0700

One of the new features in Drupal 7 is that it supports sending queries to a read-only slave database. Since version 9.0, PostgreSQL supports replication natively. In this post, I wanted to cover how to configure replication in PostgreSQL and have Drupal make use of a slave. I will use the master/slave terminology that is common in the MySQL world when referring to the master (primary) and slave (standby) servers in this post.

First, I installed PostgreSQL 9.1 on my master server along with Drupal 7.12 The steps taken to install and configure Drupal with PostgreSQL 9.1 on my master server are outlined in this gist. Then I installed PostgreSQL 9.1 on another server that will serve as a slave. My initial setup on the slave was quite simple and only involved a basic install and nothing else. The following commands were all I executed on the slave server to get a base PostgreSQL install:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.1 libpq-dev postgresql-contrib-9.1

Once the basic Drupal install was up and running on the master and the slave server has a basic PostgreSQL install, I started on configuring replication. Replication in general is documented in depth in the online PostgreSQL documentation. In this post, I will be configuring streaming replication which allows a slave server to service read queries.

The steps that need to be performed to configure streaming replication are (I will cover how to perform each step):

create a replication user for slaves to connect with
enable continuous archiving on the master
configure the master to allow remote connections with the replication user
take a base backup to be used for setting up a slave
set up a file-based log-shipping slave

The first step is to create a user for replication on the master:

sudo su postgres
psql
create role repl replication login password 'repl';

Next, the master needs to be have continuous archiving enabled. This is achieved by editing the /etc/postgresql/9.1/main/postgresql.conf file on the master and ensuring the following parameters are set:

wal_level = hot_standby
max_wal_senders = 3 # limits number of concurrent connections from standby
listen_addresses = '0.0.0.0'
archive_mode = on
archive_command = 'test ! -f /mnt/postgres/archivedir/%f && cp %p /mnt/postgres/archivedir/%f'

Now to allow remote connections for the replication user, the /etc/postgresql/9.1/main/pg_hba.conf file on the master server needs to have an entry like (this assumes the slave server I have configured has the IP address 10.39.111.10):

host  replication   repl 10.39.111.10/32      md5

Once the above modifications have been mode, we need to restart the PostgreSQL service:

sudo service postgresql restart

The master is now configured. Next we go to the slave server to take a base backup using pg_basebackup along with configuring the slave to use this base backup for its data directory:

sudo service postgresql stop
sudo mv /var/lib/postgresql/9.1/main/ /var/lib/postgresql/9.1/orig_main
sudo su postgres
pg_basebackup -D /var/lib/postgresql/9.1/main/ -P -h master_server -p 5432 -U repl -W
sudo ln -s /etc/ssl/certs/ssl-cert-snakeoil.pem /var/lib/postgresql/9.1/main/server.crt
sudo ln -s /etc/ssl/private/ssl-cert-snakeoil.key /var/lib/postgresql/9.1/main/server.key

The pg_basebackup command should result in output similar to the following:

postgres@ip-10-39-111-9:/etc/postgresql/9.1/main$ pg_basebackup -D /var/lib/postgresql/9.1/main/ -P -h 10.76.241.129 -p 5432 -U repl -W
Password: 
WARNING:  skipping special file "./server.key"
WARNING:  skipping special file "./server.crt"
WARNING:  skipping special file "./server.key"
WARNING:  skipping special file "./server.crt"
1403786/1403786 kB (100%), 1/1 tablespace
NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
postgres@ip-10-39-111-9:/etc/postgresql/9.1/main$

Next, we configure the slave to be a hot standby and to allow remote client connections (since Drupal will be connecting to the slave). This is done by editing the /etc/postgresql/9.1/main/postgresql.conf file on the slave to have the following entries:

hot_standby = on
listen_addresses = '0.0.0.0'

To allow the drupal user to connect from the master server (where apache is running), modify the /etc/postgresql/9.1/main/pg_hba.conf file on the slave (assuming 10.76.241.129 is IP address of master):

host  drupal drupal 10.76.241.129/32      md5

Next, create a recovery.conf file in the PostgreSQL data directory on the slave server:

sudo touch /var/lib/postgresql/9.1/main/recovery.conf
sudo chown postgres:postgres /var/lib/postgresql/9.1/main/recovery.conf

The following should be placed in the recovery.conf file (assuming 10.76.241.129 is IP address of master):

standby_mode = 'on'
primary_conninfo = 'host=10.76.241.129 port=5432 user=repl password=repl'

The PostgreSQL service on the slave server is now ready to be started again:

sudo service postgresql start

If everything worked correctly, log entries indicating replication is running should be present. For example, on my slave server my log file had entries like:

ubuntu@ip-10-39-111-9:/var/log/postgresql$ sudo tail -n 5 /var/log/postgresql/postgresql-9.1-main.log 
2012-07-07 22:06:50 UTC LOG:  streaming replication successfully connected to primary
2012-07-07 22:06:50 UTC LOG:  incomplete startup packet
2012-07-07 22:06:50 UTC LOG:  redo starts at 1/15000020
2012-07-07 22:06:50 UTC LOG:  consistent recovery state reached at 1/16000000
2012-07-07 22:06:50 UTC LOG:  database system is ready to accept read only connections
ubuntu@ip-10-39-111-9:/var/log/postgresql$

Now, Drupal running on the master server is ready to be configured to use a PostgreSQL slave for read-only queries! The settings.php file for the Drupal site needs to be updated to know about this slave database. My settings.php file looked like (10.39.111.10 is IP address of slave server):

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => 'localhost',
      'port' => '5432',
      'driver' => 'pgsql',
      'prefix' => '',
    ),
    'slave' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => '10.39.111.10',
      'port' => '5432',
      'driver' => 'pgsql',
      'prefix' => '',
    ),
  ),
);

I would suggest enabling query logging on the slave server so you can see read queries being sent to the slave. Query logging can be enabled by modifying the /etc/postgresql/9.1/main/postgresql.conf file to have these entries:

logging_collector = on
log_directory = 'pg_log'
log_statement = 'all'

Query log files will then be generated in the /var/lib/postgresql/9,1/main/pg_log directory.

By default, very few queries from Drupal core are sent to a slave database. The search module is probably the best module to test with to see queries being sent to the slave server. The search module can be access from your drupal site by going to http://your.ip.address/drupal/?q=search

Try searching content for a keyword. If everything is working correctly, queries should start appearing in the query log on the slave server when issuing content searches.

That’s about it for this post. Once replication is configured in PostgreSQL, having Drupal send queries to the slave is pretty straightforward.

Comparing PostgreSQL 9.1 vs. MySQL 5.6 using Drupal 7.x

Fri, 29 Jun 2012 00:00:00 -0700

Its tough to come across much information about running Drupal on PostgreSQL I find beisdes the basics of installing Drupal on PostgreSQL. In particular, I’m interested in comparisons of running Drupal on PostgreSQL versus MySQL. Previous posts such as this article from 2bits compares performance of MySQL versus PostgreSQL on Drupal 5.x and seems a bit outdated. This post from the high performance drupal group is also pretty dated and has some information with similar comparisons.

In this post, I wanted to run similar tests to what was done in the article from 2bits but on a more recent version of Drupal - 7.x. I also wanted to test out a few more complex queries that can get generated by the view module and see how they perform in MySQL versus PostgreSQL.

For this post, I used the latest GA version of PostgreSQL and for kicks, I went with an aplha release of MySQL - 5.6. I would expect to see similar results for 5.5 in tests like this. I didn’t use default configurations after installation since I didn’t see much benefit in testing that. The configurations I used for both systems are documented below.

Environment Setup

All results were gathered on EC2 instances. The base AMI used for these results is an official AMI of Ubuntu 10.04 provided by Canonical. The particular AMI used as the base image for the results gathered in this post was ami-0baf7662.

Images used were all launched in the US-EAST-1A availability zone and were large instance types. After launching this base image I installed MySQL 5.6 and Drupal 7.12. The steps I took to install these components along with the my.cnf file I used for MySQL are outlined in this gist.

The PostgreSQL 9.1 setup I performed on a separate instance along with the postgresql.conf settings I used are outlined in this gist.

APC was installed and its default configuration was used on both servers.

Data Generation

I used drush and the devel modules to generate data. I generated the following data:

users	50000
tags	1000
vocabularies	5000
menus	5000
nodes	100000
max comments per node	10

I generated this data in the MySQL installation first. The data was then migrated to the PostgreSQL instance using the dbtng_migrator module. This ensures the same data is used for all tests against MySQL and PostgreSQL. I covered how to perform this migration in a previous post.

pgbouncer

One additional setup item I performed for PostgreSQL was to install pgbouncer and configure Drupal to connect through pgbouncer instead of directly to PostgreSQL.

Installation and configuration on Ubuntu 10.04 is straightforward. The steps to install pgbouncer and the configuration I used are outlined in this gist.

The main reason for this change is the ApacheBench based test unfairly favors MySQL due to its process model. Each connection results in a new thread being spawned whereas with PostgreSQL, each new connection results in a new process being forked. The overhead of forking a new process is much larger than spawning a new thread. I did collect numbers for PostgreSQL without using pgbouncer and I do report them in the ApacheBench test section below.

pgbouncer maintains a connection pool that Drupal connects so in my settings.php file for my Drupal PostgreSQL instance, I modified my database settings to be:

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => 'localhost',
      'port' => '6432',
      'driver' => 'pgsql',
      'prefix' => '',
    ),
  ),
);

I performed this configuration step after I generated data in MySQL and migrated it to PostgreSQL.

Anonymous Users Testing with ApacheBench

First, loading the front page for each Drupal site with the devel module enabled and reporting on query execution times, the following was reported:

Database	Query Exec Times
MySQL	Executed 65 queries in 31.69 ms
PostgreSQL (with pgbouncer)	Executed 66 queries in 49.84 ms
PostgreSQL	Executed 66 queries in 95 ms

Straight out the gate, we can see there is not much difference here. 31 versus 50 ms is not going to be felt by many end users. If pgbouncer is not used, query execution time is 3 times slower though.

Next, I went to do some simple benchmarks using ApacheBench. The command used to run ab was (the number of concurrent connections, X, was the only parameter varied):

ab -c X -n 100 http://drupal.url.com/

The ab command was always run from a separate EC2 instance in the same availability zone and never on the same instance as which Drupal was running.

Results obtained with default Drupal configuration (page cache disabled) but all other caching enabled are shown in the figure below. The raw numbers are presented in the table after the figure.

Database	c = 1	c = 5	c = 10
MySQL	11.71	16.53	16.28
PostgreSQL (using pgbouncer)	8.44	11.03	11.10
PostgreSQL	4.81	7.32	7.22

The next test was run after all caches were cleared using drush. The command issued was:

drush cc

Option 1 was then chosen to clear all caches. This was done before each ab command was run. Results are shown in the figure with raw numbers presented in the table after the figure.

Database	c = 1	c = 5	c = 10
MySQL	10.50	14.08	6.28
PostgreSQL (using pgbouncer)	7.92	9.23	7.32
PostgreSQL	5	7.04	6.79

Finally, the same test was run with Drupal’s page cache enabled. Results are shown in the figure below with raw numbers presented in the table after the figure.

Database	c = 1	c = 5	c = 10
MySQL	144	282	267
PostgreSQL (using pgbouncer)	120	205	202
PostgreSQL	35	45	46

Views Queries

The views module is known to sometimes generate queries that can cause performance problems for MySQL.

Image Gallery View

The first SQL query I want to look is generated by one of the sample templates that come with the Views module. If you click ‘Add view from template’ in the Views module, by default, you will only have 1 template to choose from - the Image Gallery template. After creating a view from this template and not modifying anything about that view, I see 2 problematic queries being generated.

The first query is a query that counts the number of the rows in the result set for this view since this is a paginated view. The second query actually retrieves the results with a LIMIT clause and the appropriate OFFSET dependending on what page of the results the user is currently on. For this post, we’ll just look at the second query that retries results. That query is:

SELECT taxonomy_index.tid      AS taxonomy_index_tid, 
       taxonomy_term_data.name AS taxonomy_term_data_name, 
       Count(node.nid)         AS num_records 
FROM   node node 
       LEFT JOIN users users_node 
              ON node.uid = users_node.uid 
       LEFT JOIN field_data_field_image field_data_field_image 
              ON node.nid = field_data_field_image.entity_id 
                 AND ( field_data_field_image.entity_type = 'node' 
                       AND field_data_field_image.deleted = '0' ) 
       LEFT JOIN taxonomy_index taxonomy_index 
              ON node.nid = taxonomy_index.nid 
       LEFT JOIN taxonomy_term_data taxonomy_term_data 
              ON taxonomy_index.tid = taxonomy_term_data.tid 
WHERE  (( ( field_data_field_image.field_image_fid IS NOT NULL ) 
          AND ( node.status = '1' ) )) 
GROUP  BY taxonomy_term_data_name, 
          taxonomy_index_tid 
ORDER  BY num_records ASC 
LIMIT  24 offset 0

The response time of the query in MySQL versus PostgreSQL is shown in the figure below.

As seen in the image above, PostgreSQL can execute the query in question in 300ms or less whereas MySQL consistently takes 2800 ms to execute the query.

The MySQL execution plan looks like:

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: field_data_field_image
         type: ref
possible_keys: PRIMARY,entity_type,deleted,entity_id,field_image_fid
          key: PRIMARY
      key_len: 386
          ref: const
         rows: 19165
        Extra: Using where; Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: node
         type: eq_ref
possible_keys: PRIMARY,node_status_type
          key: PRIMARY
      key_len: 4
          ref: drupal.field_data_field_image.entity_id
         rows: 1
        Extra: Using where
*************************** 3. row ***************************
           id: 1
  select_type: SIMPLE
        table: users_node
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: drupal.node.uid
         rows: 1
        Extra: Using where; Using index
*************************** 4. row ***************************
           id: 1
  select_type: SIMPLE
        table: taxonomy_index
         type: ref
possible_keys: nid
          key: nid
      key_len: 4
          ref: drupal.field_data_field_image.entity_id
         rows: 1
        Extra: NULL
*************************** 5. row ***************************
           id: 1
  select_type: SIMPLE
        table: taxonomy_term_data
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: drupal.taxonomy_index.tid
         rows: 1
        Extra: NULL

MySQL starts from the field_date_field_image table and since there is no selective predicates in the query, chooses to scan the table using the PRIMARY key of the table. It then filters the rows scanned using the field_image_fid IS NOT NULL predicate. Since MySQL only has 1 join algorithm, nested loops, it is used to perform the remainder of the joins. A temporary table is created in memory to store the results of these joins. This is then sorted and the result set limited to the 24 requested.

The PostgreSQL execution plan looks drastically different.

 Limit  (cost=11712.83..11712.89 rows=24 width=20)
   ->  Sort  (cost=11712.83..11829.24 rows=46564 width=20)
         Sort Key: (count(node.nid))
         ->  HashAggregate  (cost=9946.90..10412.54 rows=46564 width=20)
               ->  Hash Left Join  (cost=6174.69..9597.67 rows=46564 width=20)
                     Hash Cond: (taxonomy_index.tid = taxonomy_term_data.tid)
                     ->  Hash Right Join  (cost=6140.19..8922.92 rows=46564 width=12)
                           Hash Cond: (taxonomy_index.nid = node.nid)
                           ->  Seq Scan on taxonomy_index  (cost=0.00..1510.18 rows=92218 width=16)
                           ->  Hash  (cost=5657.14..5657.14 rows=38644 width=4)
                                 ->  Hash Join  (cost=2030.71..5657.14 rows=38644 width=4)
                                       Hash Cond: (node.nid = field_data_field_image.entity_id)
                                       ->  Seq Scan on node  (cost=0.00..2187.66 rows=76533 width=8)
                                             Filter: (status = 1)
                                       ->  Hash  (cost=1547.66..1547.66 rows=38644 width=8)
                                             ->  Seq Scan on field_data_field_image  (cost=0.00..1547.66 rows=38644 width=8)
                                                   Filter: ((field_image_fid IS NOT NULL) AND ((entity_type)::text = 'node'::text) AND (deleted = 0::smallint))
                     ->  Hash  (cost=22.00..22.00 rows=1000 width=12)
                           ->  Seq Scan on taxonomy_term_data  (cost=0.00..22.00 rows=1000 width=12)

PostgreSQL has a number of other join algorithms available for use. In particular, for this query, the optimizer has decided that a hash join is the optimal choice.

PostgreSQL starts by scanning the tiny (1000 rows) taxonomy_term_data table and constructing an in-memory hash table (the build phase in a hash join). It then probes this hash table for possible matches of taxonomy_index.tid = taxonomy_term_data.tid for each row that results from a hash join of taxonomy_index and node. This hash join was a result of the field_data_field_image and node table being join with the field_data_field_image being used to build a hash table and a sequential scan of node being used to probe that hash table. Aggregation is then performed and the result set is then sorted by the aggregated value (in this case a count of node ids). Finally, the result set is limited to 24.

One neat thing about PostgreSQL is planner nodes can be disabled. So to make PostgreSQL execute the query in a similar manner to MySQL, I did:

drupal=> set enable_hashjoin=off;
SET
drupal=> set enable_hashagg=off;
SET
drupal=> set enable_mergejoin=off;
SET
drupal=>

And the execution plan PostgreSQL chose then was:

 Limit  (cost=52438.04..52438.10 rows=24 width=20)
   ->  Sort  (cost=52438.04..52552.82 rows=45913 width=20)
         Sort Key: (count(node.nid))
         ->  GroupAggregate  (cost=50237.67..51155.93 rows=45913 width=20)
               ->  Sort  (cost=50237.67..50352.45 rows=45913 width=20)
                     Sort Key: taxonomy_term_data.name, taxonomy_index.tid
                     ->  Nested Loop Left Join  (cost=0.00..46682.48 rows=45913 width=20)
                           ->  Nested Loop Left Join  (cost=0.00..33783.81 rows=45913 width=12)
                                 ->  Nested Loop  (cost=0.00..18575.38 rows=38644 width=4)
                                       ->  Seq Scan on field_data_field_image  (cost=0.00..1547.66 rows=38644 width=8)
                                             Filter: ((field_image_fid IS NOT NULL) AND ((entity_type)::text = 'node'::text) AND (deleted = 0::smallint))
                                       ->  Index Scan using node_pkey on node  (cost=0.00..0.43 rows=1 width=8)
                                             Index Cond: (nid = field_data_field_image.entity_id)
                                             Filter: (status = 1)
                                 ->  Index Scan using taxonomy_index_nid_idx on taxonomy_index  (cost=0.00..0.36 rows=3 width=16)
                                       Index Cond: (node.nid = nid)
                           ->  Index Scan using taxonomy_term_data_pkey on taxonomy_term_data  (cost=0.00..0.27 rows=1 width=12)
                                 Index Cond: (taxonomy_index.tid = tid)

The above plan takes 2 seconds to execute against PostgreSQL. You can see it is very similar to the MySQL plan. It starts with the field_data_field_image table and performs nested loop joins to join the remainder of the tables. In this case, a sort must be performed before the aggregation that is expensive to perform. Using the HashAggregate operator in PostgreSQL would greatly reduce that cost.

So you can see out of the box, PostgreSQL performs much better on this query.

Simple View

I created a simple view that filters and sorts on content criteria. A screenshot of my view construction page can be seen here.

The resulting SQL query that gets executed by this view is:

SELECT DISTINCT node.title                            AS node_title, 
                node.nid                              AS nid, 
                node_comment_statistics.comment_count AS 
                node_comment_statistics_comment_count, 
                node.created                          AS node_created 
FROM   node node 
       INNER JOIN node_comment_statistics node_comment_statistics 
         ON node.nid = node_comment_statistics.nid 
WHERE  (( ( node.status = '1' ) 
          AND ( node.comment IN ( '2' ) ) 
          AND ( node.nid >= '111' ) 
          AND ( node_comment_statistics.comment_count >= '2' ) ))
ORDER  BY node_created ASC 
LIMIT  50 offset 0

The response time of the query in MySQL versus PostgreSQL is shown in the figure below.

As seen in the image above, PostgreSQL can execute the query in question in 200ms or less whereas MySQL can take up to 1000 ms to execute the query.

The MySQL execution plan looks like:

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: node
         type: index
possible_keys: PRIMARY,node_status_type
          key: node_created
      key_len: 4
          ref: NULL
         rows: 100
        Extra: Using where; Using temporary
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: node_comment_statistics
         type: eq_ref
possible_keys: PRIMARY,comment_count
          key: PRIMARY
      key_len: 4
          ref: drupal.node.nid
         rows: 1
        Extra: Using where

MySQL chooses to start from the node table and scans an index on the created column. A temporary table is then created in memory to store the results of this index scan. The items stored in the temporary table are then processed to eliminate duplicates (for the DISTINCT). For each distinct row in the temporary table, MySQL then performs a join to the node_comment_statistics table by performing an index lookup using its primary key.

The PostgreSQL execution plan for this query looks like:

 Limit  (cost=6207.15..6207.27 rows=50 width=42)
   ->  Sort  (cost=6207.15..6250.75 rows=17441 width=42)
         Sort Key: node.created
         ->  HashAggregate  (cost=5453.36..5627.77 rows=17441 width=42)
               ->  Hash Join  (cost=1985.31..5278.95 rows=17441 width=42)
                     Hash Cond: (node.nid = node_comment_statistics.nid)
                     ->  Seq Scan on node  (cost=0.00..2589.32 rows=38539 width=34)
                           Filter: ((nid >= 111) AND (status = 1) AND (comment = 2))
                     ->  Hash  (cost=1546.22..1546.22 rows=35127 width=16)
                           ->  Seq Scan on node_comment_statistics  (cost=0.00..1546.22 rows=35127 width=16)
                                 Filter: (comment_count >= 2::bigint)

PostgreSQL chooses to start by scanning the node_comment_statistics table and building an in-memory hash table. This hash table is then probed for possible mathces of node.nid = node_comment_statistics.nid for each row that results from a sequential scan of the node table. The result of this hash join is then aggregated (for the DISTINCT) before being sorted and limited to 50 rows.

Its worth noting that with out of the box settings, the above query would do a disk based sort (sort method is viewable using EXPLAIN ANALYZE in PostgreSQL). When doing a disk based sort, the query takes about 450 ms to execute. I was running all my tests with work_mem set to 4MB though which results in a top-N heapsort being used.

Conclusion

In my opinion, the only issue with using PostgreSQL as your Drupal database is that some contributed modules will not work out of the box with that configuration.

Certainly, from a performance point of view, I see no issues with using PostgreSQL with Drupal. In fact, for Drupal sites using the Views module (probably the majority), I would say PostgreSQL is probably even a better option than MySQL due to its more advanced optimizer and execution engine. This does assume pgbouncer is being used and Drupal is not connecting directly to PostgreSQL. Users who do not use pgbouncer and perform simple benchmarks like the ones I did with ab are likely to see poor performance against PostgreSQL.

I’m working a lot with Drupal on PostgreSQL these days. I’ll be sure to share any interesting experiences I have here.

Migrating Drupal 7 Site from MySQL to PostgreSQL on Ubuntu 10.04

Tue, 26 Jun 2012 00:00:00 -0700

I recently needed to migrate a Drupal 7 site running on a MySQL 5.5 database to a PostgreSQL 9.1 database. This brief post describes the steps I took to achieve this. The steps outlined here were only tested on Ubuntu 10.04

First, I installed a fresh copy of PostgreSQL 9.1.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.1 libpq-dev

After the installation is complete, a schema and user account is created for Drupal.

sudo su postgres
createuser -D -A -P drupal
createdb --encoding=UTF8 -O drupal drupal
exit

The above creates a user account named drupal (you will be prompted for a password for the user account when running the command) and a schema named drupal.

Next, PostgreSQL needs to be configured to allow connections from Apache for Drupal. This is done by modifying the /etc/postgresql/9.1/main/pg_hba.conf file. The following line needs to be commented out or deleted:

local   all             all                                     peer

The line to added in this file is:

host    drupal          drupal          127.0.0.1/32            password

After this file is modified, PostgreSQL needs to be restarted.

sudo service postgresql restart

For the migration, we are going to assume drush is installed on the server we will be performing the migration. We are also going to assume MySQL and PostgreSQL are running on the same server although this is certainly not a requirement for these instructions.

The module that performs the real work of the migration is the dbtng_migrator module. This module is installed in the same manner as any other Drupal module. After the module is installed, the settings.php file for your drupal installation then needs to be modified to point to your source and destination database. In my case, I updated my settings.php file to look like:

$databases = array (
  'default' => array (
    'default' =>
      array (
        'database' => 'drupal',
        'username' => 'drupal',
        'password' => 'drupal',
        'host' => 'localhost',
        'port' => '',
        'driver' => 'mysql',
        'prefix' => '',
      ),
  ),
  'dest' => array (
    'default' =>
      array (
        'database' => 'drupal',
        'username' => 'drupal',
        'password' => 'drupal',
        'host' => 'localhost',
        'port' =>'',
        'driver' => 'pgsql',
        'prefix' =>'',
      ),
    ),
);

As you can see in my case, the default schema that I am currently running on is a MySQL database and I am planning on migrating to a PostgreSQL database running on the same machine.

Now, to perform the migration from the command line using drush, its as simple as:

drush cache-clear drush
drush dbtng-replicate default dest

When the migration finishes, output similar to the following will be seen (this is just a small portion of the output):

$ drush dbtng-replicate default dest
...
cache_update successfully migrated.                    [status]
authmap successfully migrated.                         [status]
role_permission successfully migrated.                 [status]
role successfully migrated.                            [status]
users successfully migrated.                           [status]
users_roles successfully migrated.                     [status]
$

Finally, after the database migration is successfully completed, the settings.php file needs to be updated to point to the new database. In my case, the database settings after my migration looked like:

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => 'drupal',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'pgsql',
      'prefix' => '',
    ),
  ),
);

That was it for my migration. Granted, I had a small drupal site to migrate and the only additional modules I had installed were the views and devel modules so I did not need to worry about contributed modules working with the PostgreSQL database. Next step would be to be configure PostgreSQL in a more optimal which I did not go in to here.

Using Akiban Server with Drupal 6

Fri, 18 May 2012 00:00:00 -0700

In my previous post, I mentioned I’m working on a database driver for Drupal 7 for the Akiban Server. However, we have some clients who use Drupal 6 so I wanted to talk about how we work with those clients in this post.

Drupal 6 does not have a database abstraction layer so it is not as easy to integrate Akiban in this case. With Drupal 6, we do not attempt to run all of Drupal on Akiban. What we have done with our existing customers who use Drupal 6 is to send certain problem queries to Akiban (running in our MySQL replication configuration) and everything else to MySQL. The Akiban Server speaks the PostgreSQL protocol as I discussed before. Hence, for Drupal 6, we use the PostgreSQL database driver to talk to Akiban.

Since Drupal 6 does not support speaking to multiple different database backends at the same time, we apply a patch to get started. Basically, this patch allows connections to be open to both an existing MySQL server and the Akiban Server at the same time. It could be used to do the same with a PostgreSQL database as well.

Once that patch is applied, to send a query to Akiban, the active database connection is set to Akiban and a query is fired off for the Akiban Server to execute. The following code snippet shows an example.

if ($this->use_akiban) {
   db_set_active('akiban');
   $result = db_query_range($query, 
                            $args, 
                            $offset, 
                            $this->pager['items_per_page']);
}
db_set_active('default')
if (! $result) { /* if Akiban failed go to regular MySQL */
   $result = db_query_range($query, 
                            $args, 
                            $offset, 
                            $this->pager['items_per_page']);
}

As can be seen in the code above, its also possible to detect if the query failed against Akiban and re-issue it against MySQL.

When deployed in this configuration, connection details for the Akiban Server is specified in the settings.php file for a Drupal site.

$db_url = array(
  "default" => "mysql://drupal:drupal@mysql_hostname/drupal_schema",
  "akiban"  => "pgsql://drupal:drupal@akiban_hostname:15432/drupal_schema"
);

We’ve also done some work with clients where we have made patches to the Views module for Drupal. These patches allow a client to send the queries generated by a specific view to an Akiban Server.

Taken together, this makes it quite easy for us to work with Drupal 6 and send problematic queries to Akiban.

Akiban Server Progress with Drupal 7

Mon, 14 May 2012 00:00:00 -0700

The call for papers for DrupalCon Munich closed on Friday and I submitted a session related to the work I’m doing on developing a database module for the Akiban Server with Drupal 7. That work has not been open sourced yet but will be before August. We also plan on open sourcing and releasing the Akiban Server for public download by August as well. The end result of this work will be a database driver for the Akiban Server that will allow Drupal 7 to run on Akiban.

In this post, I wanted to briefly show the type of results I’ve been seeing from running Drupal on Akiban. To do this, I constructed a simple view using the Views module and benchmarked the query that resulted from this view.

Environment Setup

I also created an AMI from the running instance after all the steps outlined were performed. This AMI has MySQL 5.6 installed along with Drupal 7.12 and data generated with drush.

The Akiban AMI cannot be made available for general download yet since we have not open-sourced our stack as of this time. Once our stack has been open-sourced I will update this post with a link to an AMI that can be downloaded. However, if you are interested in seeing the results here for yourself, feel free to contact me and I should be able to grant access to an EC2 instance for testing.

Data Generation

I used drush and the devel modules to generate data so the view would be operating on some data. I generated the following data:

users	50000
tags	1000
vocabularies	5000
menus	5000
nodes	100000
max comments per node	10

View and SQL Query

I created a simple view that filters and sorts on content criteria. A screenshot of my view construction page can be seen here.

The resulting SQL query that gets executed by this view is:

SELECT DISTINCT node.title                            AS node_title, 
                node.nid                              AS nid, 
                node_comment_statistics.comment_count AS 
                node_comment_statistics_comment_count, 
                node.created                          AS node_created 
FROM   node node 
       INNER JOIN node_comment_statistics node_comment_statistics 
         ON node.nid = node_comment_statistics.nid 
WHERE  (( ( node.status = '1' ) 
          AND ( node.comment IN ( '2' ) ) 
          AND ( node.nid >= '111' ) 
          AND ( node_comment_statistics.comment_count >= '2' ) ))
ORDER  BY node_created ASC 
LIMIT  50 offset 0

Performance Comparison

The response time of the query in Akiban versus MySQL is shown below.

As seen in the image above, Akiban can execute the query in question in 5 ms or less whereas MySQL consistently takes 1200 ms to execute the query. In the next section I’ll go into details of how Akiban executes this query.

Secondly, numbers were obtained using the mysqlslap benchmark tool from MySQL to demonstrate how Akiban performs versus MySQL with varying degrees of concurrency.

MySQL Execution Plan

Using Maatkit to visualize the MySQL execution plan, we get:

JOIN
+- Filter with WHERE
|  +- Bookmark lookup
|     +- Table
|     |  table          node_comment_statistics
|     |  possible_keys  PRIMARY,comment_count
|     +- Unique index lookup
|        key            node_comment_statistics->PRIMARY
|        possible_keys  PRIMARY,comment_count
|        key_len        4
|        ref            drupal.node.nid
|        rows           1
+- Table scan
   +- TEMPORARY
      table          temporary(node)
      +- Filter with WHERE
         +- Bookmark lookup
            +- Table
            |  table          node
            |  possible_keys  PRIMARY,node_status_type
            +- Index scan
               key            node->node_created
               possible_keys  PRIMARY,node_status_type
               key_len        4
               rows           100

MySQL chooses to start from the node table and scans an index on the created column. A temporary table is then created in memory to store the results of this index scan. The items stored in the temporary table are then processed to eliminate duplicates (for the DISTINCT). For each distinct row in the temporary table, MySQL then performs a join to the node_comment_statistics table by performing an index lookup using its primary key.

Akiban Execution Plan

The tables involved in the query fall into a single table group in Akiban - the node group. Grouping is explained by our CTO in this post and that post includes a grouping for Drupal where you can see the node group. For this query, it means all joins within the node group are executed with essentially zero cost. It also allows for the creation of Akiban group indexes. A group index is an index that can span multiple tables along a single branch within a table group.

A covering group index for this query is:

CREATE INDEX cvr_gi ON node
(
  node.status,
  node.comment,
  node.created,
  node.nid,
  node_comment_statistics.comment_count,
  node_comment_statistics.nid,
  node.title
) USING LEFT JOIN

Notice that the node.created column is included in this index so a sort could be avoided.

The other large advantage Akiban brings when executing this query is the query optimizer is intelligent enough to determine that the DISTINCT is not required in the query due to the 1-to-1 mapping between node and node_comment_statistics and the fact that an INNER JOIN is being performed between these 2 tables.

Limit_Default(limit=50: project([Field(6), Field(3), Field(4), Field(2)]))
  project([Field(6), Field(3), Field(4), Field(2)])
    Select_HKeyOrdered(Index(cvr_gi(BoolLogic(AND -> Field(3) >= Literal(111), Field(4) >= Literal(2) -> BOOL))
      IndexScan_Default(Index(cvr_gi(>=UnboundExpressions[Literal(1), Literal(2)],<=UnboundExpressions[Literal(1), Literal(2)]))

The above execution plan is in the Akiban format. In this format, you read the plan like a tree so we start from the leaf nodes. The above plan starts with a scan of the cvr_gi index using the node.status and node.comment predicates. It then filters rows from this scan (the Select_HKeyOrdered operator performs this filtering) before limiting the results to the size of the result set requested.

Conclusion

To wrap up, I briefly showed some of the performance benefits we are seeing when running Drupal 7 on the Akiban Server. In the not too distant future, we will be open sourcing our stack here at Akiban and providing downloads of the Akiban Server. I will also be making the database driver for the Akiban Server for Drupal 7 available for download on drupal.org once its complete.

If you are interested in trying this out yourself or want to verify the results before this work becomes publically available, feel free to contact me and I should be able to set you up with access to an EC2 instance so you try if for yourself.