Blue smoke from the chimney, I repeat blue smoke from the chimney! #Drupal8 release candidate 1 has been released! Good day for #Drupal!

— Marc Drummond (@MarcDrummond) October 7, 2015

That’s right, Drupal 8 has it’s first release. But what does that mean? Is it done? Can I start using it yet? What kind of changes are coming? Will dawehner get to sleep, at last?

Are we there yet?

Despite all the rejoicing on social media, this isn’t the final release for Drupal 8 – it’s only the first Release Candidate. This means that we have (officially!) 0 “critical” bugs left to fix in Drupal 8. That means exactly what it sounds like: there are no critical, world-ending bugs left… that we know of. Just like any software product, we’ll continue to discover critical issues through its entire life cycle. We’re still finding occasional critical issues in Drupal 7 almost five years since its first release candidate; that’s just a part of supporting a piece of software over the long term. The RC phase means that while Drupal 8 is stable enough to use, we’re still discovering critical bugs a little too frequently to recommend it for everyone, in every use case.

“A little too frequently” means that the rate of critical bugs incoming is still too high to be able to promise the fast respond-and-fix turnaround that we want. Every two weeks we’ll create a new Release Candidate version that fixes whatever new criticals have been discovered. Once the core team is confident that they can squash bugs in a timely enough manner, they’ll (finally) release Drupal version 8.0.0.

But when will it REALLY be released?

“When it’s ready” still applies! But we are very, very close now. To give you a point of reference, Drupal 7 went through four Release Candidates before release (two months). That codebase was a lot more fragile than this one, so it’s reasonable to hope that we’ll have a very Drupally Christmas season this year. Personally I’m betting on January.

Can I use it yet?

Yes! Some terms and conditions apply.

Just because there are no criticals left, doesn’t mean that D8 is completely bug-free! We have a big pile of known “major” issues that have been deferred until after 8.0.0, which should impact your decision. You can see at that link that some of them are already ready to be committed. The catch is that during the RC phase, we aren’t allowed to commit these fixes. We’re basically only allowed to work on criticals and documentation. So there are still some serious issues that might be a problem in some use cases.

The biggest issue (that I know of) is a potential incompatibility between Drupal 8’s new “cache tags” header and some hosting providers. The problem is that Drupal includes some important caching information on the “back of the envelope” of its response to a page request, and it’s possible to run out of envelope! If the cache tags header gets too long for the web host to handle, it can behave unpredictably. You might get white screens of death, or it might just shorten the cache tags header, removing important information. There’s a solution in the works to allow a maximum length setting, but it won’t make it in until 8.0.1 (two weeks after 8.0.0). In the meantime you should avoid D8 if you have any very complex pages with many elements. The examples in that ticket are good ones: a news site with very complex layouts, or a single page site with a lot of “stuff” crammed onto the one, front page.

The other “gotcha” to bear in mind is that it will take some time for Drupal’s contributed modules ecosystem to catch up with the new version. According to Bluespark’s status of the top 100 modules for D8 page, so far only 9 of the top 100 D7 modules have a D8 release marked “stable.” 19 of those top 100 modules are included in D8 core however, so our total count is up to 28. This is enough to give a good foundation for relatively simple sites, especially if you have some PHP skills under your belt. But I wouldn’t go building a complex Intranet on it just yet!

Wait, so it’s still busted?

No! Drupal 8 is a solid platform for most use cases – that’s the point of the RC release! It’s time to go ahead and use it for site builds. Just take it easy and use it for simple sites, first. Give the rest of the community a chance to release stable modules, and hold off on that Facebook-buster behemoth website you’ve got planned until a few months after launch.

What happens after 8.0.0?

After 8.0.0 is released, we will make an enormous, fundamental shift in how Drupal is developed. We will start using semantic versioning with a regular release schedule. Every two weeks we’ll release a new “patch level’ release: 8.0.1, 8.0.2, and so on. Patch level releases will be bug fixes only, and will be backwards-compatible – that means they won’t break anything on your site. Approximately every 6 months, we’ll release a new “minor level” release: 8.1.0, 8.2.0, etc. Minor releases are allowed to contain new features, but they are still guaranteed to be backwards-compatible. So even these releases won’t break anything on your site. We’re still figuring out the exact process for minor releases, but they will include similar phases to what we’ve seen with D8 core: a beta phase, and release candidates until we’re sure there are no more criticals.

What about API changes, and features that would break existing sites? We won’t even start developing on those until well into the D8 life cycle. Those changes will belong in the 9.x branch, and will be kept completely separate from anything that could touch your site.

The key take-away here is that D8 updates should never break your site. They may add features, but they will not interfere with whatever you’ve already built. We’ll continue a regular pace of improving the product in a predictable, scheduled, and backwards-compatible way.

Where are the best Drupal 8 release parties?

The Drupal Association is coordinating promotion for official Drupal 8 launch parties. If you want to host one, just fill out their form and they’ll help you promote it! So far no one has built a site mapping the parties, but keep an eye out in the #drupal hashtag on twitter!

Who do I congratulate? Who do I thank?

Drupal 8 RC 1 is the combined effort of more than 3200 contributors. That is an incredible number. By comparison, Apache, the world’s most popular open source webserver, has 118 contributors. MySQL, the database platform which runs an enormous portion of the Internet, has 1320 contributors. So you can basically walk up to anyone at a Drupalcon and thank him or her!

Most of the contributors to Drupal 8 leaned on the support, training, and hand-holding of mentors at Drupal events all over the world. I know I needed a mentor for my first core contributions, and I got to turn around and mentor other people myself. The mentors are the support network that made this level of mass contribution possible.

But the level of effort is definitely not evenly distributed. Most contributors have made fewer than 20 registered contributions. But some people have really gone above and beyond what anyone would expect. It’s no exaggeration to say that these people have shaped the future of the Internet.

It is easy to concentrate on the number of contributions as the only metric of involvement in the release of D8. But some of the biggest influences on Drupal 8 have been community leaders, whose effort is not counted in commits under their own names. The initiative leads who architected and directed all this contribution: heyrocker, Senpai, jlambert, Crell, dmitrig01, Gábor Hojtsy, Jose Reyero, mitchell, jenlampton, bleen18, jackalope, ericduran, jhood, jacine, shyamala, rupl, JohnAlbin, twom, and sofiya. Without them, we would have had nothing to commit!

Listing all of those names brings to mind the platform that they all use to contribute and coordinate: drupal.org, maintained by the Drupal Association. It also brings to mind the events, like Drupalcon, Drupalcamps, Dev Days, which everyone attends to collaborate, teach, and learn; also maintained by the Drupal Association. Not to mention the Drupal 8 Accelerate program, which raised $250,000 towards developer grants; also created and maintained by the Drupal Association. The people at the Association have worked tirelessly to support this release.

All of this developer time is extremely valuable, and not all of it came out of the developers’ own free time. Huge swaths of Drupal 8 development have been sponsored by the companies that participate in the community. We’ve only been tracking their contributions for a short time, but the information we have is powerful. This release would not have happened without the developer time donated by companies like Acquia, MD Systems, Chapter Three, Tag1, and Druid. A quick glance at Drupal.org’s Drupal Services page shows us that contribution is a normal part of the culture for the biggest Drupal companies. These were the top 5, but almost every major Drupal shop has contributed in some measure. Thank you to these companies for believing in our product and supporting it so generously.

Finally, the people who bear the greatest personal responsibility are definitely the core maintainers. These people don’t just deserve your thanks; they deserve lifetime supplies of free beer sent to their homes. I can’t offer that on a blog; all I can say is THANK YOU.

Alex Bronstein

Dries Buytaert

Angie “webchick” Byron

Nat Catchpole

Jess Myrbo

Alex Pott

To everyone who contributed, but especially the people I’ve listed here: You’ve made a new generation of Internet innovation possible. Thank you.

Migrate in Drupal 7 is a fantastic piece of code. It is not designed to be used from the GUI, rather, it provides a framework of “source”, “destination”, and “migration” classes so that even the most convoluted migration is 90% written for you. To create a migration in Drupal 7, you create a custom module, declare your migrations in a hook_info, and then extend the built in “migration” class. You instantiate one of the given classes for the source material (is it a CSV? JSON? Direct connection to a custom DB?), then one of the classes for the destination (is it a content type? Taxonomy term?). Then you add one simple line of code mapping each field from source to destination. If you know what you’re doing, the task I had in mind shouldn’t take more than 15 minutes per source.

It’s not quite so easy in Drupal 8. First of all, with Migrate in core, we had to greatly simplify the goals for the module. The version of Migrate that is really functional and stable is specifically and only the basic framework. There is a separate migrate_drupal module to provide everything you need for migrating from Drupal 6 or 7. This has been a laser-tight focus on just the essentials, which means there’s no UI, very little drush support, and definitely no nice extras like the ability to read non-Drupal sources.

My task this week became to write the first CSV source for Drupal 8 Migrate.

Drupal 8 Migrate Overview

Drupal 8 Migrations, when you’re using classes that already exist, are actually even easier than Migrate 7. All you do is write a single YAML file for each kind of data you’re transferring, and put it in a custom module’s config/install directory. Your YAML file gives your migration a name and a group, tells us what the source is for data, maps source fields to destination fields, and tells us what the destination objects are. Here’s an example Migration definition file from core. See if you can understand what’s being migrated here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

id: d6_system_site
label: Drupal 6 site configuration
migration_groups:
  - Drupal 6
source:
  plugin: variable
  variables:
    - site_name
    - site_mail
    - site_slogan
    - site_frontpage
    - site_403
    - site_404
    - drupal_weight_select_max
    - admin_compact_mode
process:
  name: site_name
  mail: site_mail
  slogan: site_slogan
  'page/front': site_frontpage
  'page/403': site_403
  'page/404': site_404
  weight_select_max: drupal_weight_select_max
  admin_compact_mode: admin_compact_mode
destination:
  plugin: config
  config_name: system.site

You probably figured it out: this migration takes the system settings (variables) from a Drupal 6 site, and puts them into the Drupal 7 configuration. Not terribly hard, right? You can even do data transformations from the source field value to the destination.

Unfortunately, the only sources we have so far are for Drupal 6 and 7 sites, pulling directly from the database. If you want to use Migrate 8 the way we used Migrate 7, as an easy way to pull in data from arbitrary sources, you’ll have to contribute.

Writing a source plugin in Migrate_plus

Enter Migrate Plus module. This is the place in contrib, where we can fill out all the rest of the behavior we want from Migrate, that’s not necessarily a core requirement. This is where we’ll be writing our source plugin.

To add a source plugin, just create a .php file in migrate_plus/src/Plugins/migrate/source . Drupal will discover the new plugin automatically the next time you rebuild the cache. The filename has to be the same as the name of the class, so choose carefully! My file is called CSV.php . Here’s the top of the file you need for a basic :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?php
/**
 * @file
 * Contains \Drupal\migrate_plus\Plugin\migrate\source\csv.
 */

namespace Drupal\migrate_plus\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SourcePluginBase;

/**
 * Source for CSV files.
 *
 * @MigrateSource(
 *   id = "csv"
 * )
 */
class CSV extends SourcePluginBase {

I’m calling this out separately because for newbies to Drupal 8, this is the hard part. This is all the information that Drupal needs to be able to find your class when it needs it. The @file comment is important. That and the namespace below have to match the actual location of the .php file.

Then you declare any other classes that you need, with their full namespace. To start with all you need is SourcePluginBase.

Finally you have to annotate the class with that @MigrateSource(id=“csv”). This is how Migrate module knows that this is a MigrateSource, and the name of your Plugin. Don’t miss it!

Inside the class, you must have the following methods. I’ll explain a bit more about each afterwards.

• initializeIterator() : Should return a valid Iterator object.
• getIds() : Should return an array that defines the unique identifiers of your data source.
• __toString() : Should return a simple, string representation of the source.
• fields() : Should return a definitive list of fields in the source.
• __construct() : You don’t NEED this method, but you probably will end up using it.

initializeIterator()

An Iterator is a complicated sounding word for an Object that contains everything you need to read from a data source, and go through it one line at a time. Maybe you’re used to fopen(‘path/to/file’, ‘r’) to open a file, and then you write code for every possible operation with that file. An iterator takes care of all that for you. In the case of most file-based sources, you can just use the SplFileObject class that comes with PHP.

Any arguments that were passed in the source: section of the YAML file will be available under $this->configuration. So if my YAML looks like this:

1
2
3

source:
  plugin: csv
  path: '/vagrant/import/ACS_13_1YR_B28002_with_ann.csv'

My initializeIterator( ) method can look like this:

1
2
3
4
5
6

public function initializeIterator() {
  // File handler using our custom header-rows-respecting extension of SPLFileObject.
  $file = new SplFileObject($this->configuration['path']);
  $file->setFlags(SplFileObject::READ_CSV);
  return $file;
}

Not too complicated, right? This method is called right at the beginning of the migration, the first time Migrate wants to get any information out of your source. The iterator will be stored in $this->iterator.

getIds()

This method should return an array of all the unique keys for your source. A unique key is some value that’s unique for that row in the source material. Sometimes there’s more than one, which is why this is an array. Each key field name is also an array, with a child “type” declaration. This is hard to explain in English, but easy to show in code:

1
2
3
4
5
6
7

public function getIDs() {
  $ids = array();
  foreach ($this->configuration['keys'] as $key) {
    $ids[$key]['type'] = 'string';
  }
  return $ids;
}

We rely on the YAML author to tell us the key fields in the CSV, and we just reformat them accordingly. Type can be ‘string’, ‘float’, ‘integer’, whatever makes sense.

__toString()

This method has to return a simple string explanation of the source query. In the case of a file-based source, it makes sense to print the path to the file, like this:

1
2
3

public function __toString() {
  return (string) $this->query;
}

fields()

This method returns an array of available fields on the source. The keys should be the machine names, the values are descriptive, human-readable names. In the case of the CSV source, we look for headers at the top of the CSV file and build the array that way.

__construct()

The constructor method is called whenever a class is instantiated. You don’t technically HAVE to have a constructor on your source class, but you’ll find it helpful. For the CSV source, I used the constructor to make sure we have all the configuration that we need. Then I try and set sane values for fields, based on any header in the file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
  parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);

  // Path is required.
  if (empty($this->configuration['path'])) {
    return new MigrateException('You must declare the "path" to the source CSV file in your source settings.');
  }

  // Key field(s) are required
  if (empty($this->configuration['keys'])) {
    return new MigrateException('You must declare the "keys" the source CSV file in your source settings.');
  }

  // Set header rows from the migrate configuration.
  $this->headerRows = !empty($this->configuration['header_rows']) ? $this->configuration['header_rows'] : 0;

  // Figure out what CSV columns we have.
  // One can either pass in an explicit list of column names to use, or if we have
  // a header row we can use the names from that
  if ($this->headerRows && empty($this->configuration['csvColumns'])) {
    $this->csvColumns = array();

    // Skip all but the last header
    for ($i = 0; $i < $this->headerRows - 1; $i++) {
      $this->getNextLine();
    }

    $row = $this->getNextLine();
    foreach ($row as $key => $header) {
      $header = trim($header);
      $this->getIterator()->csvColumns[] = array($header, $header);
    }
  }
  elseif ($this->configuration['csvColumns']) {
    $this->getIterator()->csvColumns = $this->configuration['csvColumns'];
  }
}

Profit!

That’s it! Four simple methods, and you have a new source type for Drupal 8 Migrate. Of course, you will probably find complications that require a bit more work. For CSV, supporting a header row turned out to be a real pain. Any time Migrate tried to “rewind” the source back to the first line, it would try and migrate the header row! I ended up extending SplFileObject just to handle this issue.

Here’s the YAML file I used to test this, importing a list of states from some US Census data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

id: states
label: States
migration_groups:
  - US Census

source:
  plugin: csv
  path: '/vagrant/import/ACS_13_1YR_B28002_with_ann.csv'
  header_rows: 2
  fields:
    - Id2
    - Geography
  keys:
    - Id2

process:
  name: Geography
  vid:
    -
      plugin: default_value
      default_value: state

destination:
  plugin: entity:taxonomy_term

You can see my CSV source and Iterator in the issue queue for migrate_plus. Good luck, and happy migrating!

Thanks

I learned a lot this week. Big thanks to the Migrate Documentation, but especially to chx, mikeryan, and the other good folks in #drupal-migrate who helped set me straight.

This issue should not reflect badly on the Drupal community, or the Drupal product at all. Vulnerabilities happen to every software project – particularly the large and complex ones like Drupal! In this case it was the result of a choice in the database abstraction layer to use emulated prepared statements. There’s a great dissection of the whole vulnerability at ircmaxell, but the point here is that it was an intentional decision. We were aware of a theoretical security risk, just as we are in making lots of decisions. But theoretical risks don’t mean much compared with real, measurable losses from the available alternatives. As I said before, this can happen to any software project, and Drupal is a relatively responsible, well written one. What’s interesting now, is the response.

First of all, I am amazed to read responses from many Drupal users who are panicked at having to run a diff of their sites, because they don’t have appropriate tools in place. If you are developing without a VCS and automated backups, you are doing more harm than good. Just stop. Take a week to learn the basic requirements of a development environment, and start employing them. End of story.

I’m not concerned for those people – their sites were disasters waiting for an excuse anyway. What’s frightening about this particular situation is that even if you are working with backups and a VCS, even if you patch critical security vulnerabilities on an aggressive schedule, it still wasn’t good enough.

All of my Drupal 7 sites are affected by this PSA. I work for a large, well-respected agency, with access to leading-edge workflows and tools. We follow best practices. All of my sites are secured well beyond PCI requirements. But PCI requirements say that critical security patches have to be applied within 30 days of release. Our best practices include patch review from the tech lead, and validating patches on test environments before pushing them live. With only 7 hours between patch release and exploits in the wild, there isn’t time for any of that.

I’ve heard people complain that it’s too difficult to update Drupal. “drush up —security-only” seems pretty simple to me, or at least simple enough that further simplification won’t address the real problem. That’s because the real problem isn’t that it’s difficult to apply updates – it’s that a human be ing has to initiate them. I live in the Central European timezone, GMT+6. The patch was released at 10PM for me, and bots were exploiting it by 5AM the following morning. I went to work that day and initiated the patching process, so that my patches could be “responsibly” deployed to live with 24-48 hours of client validation time on my dev and staging environments. Despite being relatively on top of patches and responding relatively quickly, the fact that I’m human, and my clients are human, meant we never stood a chance of patching this issue fast enough. Even if we skipped validation, and even if the update process was just one button (rather than two commands), we would still have failed to update in time. I find myself reminded of the Battlestar Galactica pilot, where the Cylon robots are chasing the humans. After each hyperspace jump, the humans have 33 minutes to complete the calculations for another jump before the machines catch up with them. After 130 hours and 237 jumps, it becomes apparent that the humans’ need for sleep is a critical vulnerability.

The only solution is automated patching. It’s hard to figure out a workflow that allows it; indeed you’re forced into post-hoc testing, which means engineering an easy rollback solution. The truth is that 99% of security patches will not affect any of the functionality you’ve customized or upon which you rely, so hopefully this will be an edge case. But it’s a problem that actually has to be addressed. Here’s how I’m adapting my own projects over the coming weeks:

Every wednesday, every hour, my Jenkins instance runs a script which checks modules and core for each project for security updates. When an update is available, it automatically creates a branch off of Stable (my staging branch), applies the updates, and pushes the result up to the server. My git scripts already create a new subdirectory environment for every pushed branch. Once the environment is ready, Jenkins runs all available behat tests against the new branch. If all tests pass, the branch is automatically merged back into Master, Stage, and Live, and pushed. This push operation triggers a normal Jenkins deployment, which takes a backup anyway. An email is generated to the project administrator advising them which security updates were automatically applied, and linking to the relevant changefiles.

I’m excited about implementing this new layer of automation, because it builds on the best practice workflows I already like (test driven development, git flow VCS organization, automated deployment and backups…) to produce a tangible time savings and security improvement for my sites. At the same time, I can’t say that this is something I recommend for EVERYONE, precisely because it requires such a high level of environment maintenance. When you’re a one-person development shop, it’s hard to afford the time to set up the perfect development environment. It’s hard to convince those bottom-of-the-food-chain clients to pay for things like automated testing and deployment. And certainly once you have those things set up, you don’t get paid for maintaining them!

I’m going to be keeping my eyes open for better solutions that can be applied by the “developer on the street.” Something relatively easy, but which allows the same kind of automated, fast response time for security patches. I’m interested in any ideas you want to post in the comments!

For now, the only solution is a slow one – we clear static entity caches when we generate multilingual titles. That’s not an awesome fix, but it’s hard to think of a better one without any of the D8 cache tagging functionality. Massive kudos to bburg for figuring this out!

And for those of you keeping score, this is a good example of how to file a bug report for a really complex issue in a really popular module… and follow up until you resolve it.

If you haven’t been to a code sprint before, it’s basically a coding party. Developers get together and help each other contribute better and faster by reviewing code on the spot, mentoring each other, and generally working in small ad-hoc groups. It’s a lot of fun, and gives a big boost to development of the next generation of Drupal.

Forum One will provide the locale in downtown DC complete with pizza, beer, and soda. We also have a few of our core mentors on hand to help you get started if this is your first time contributing to core. Because of the building security, if you want to attend you have to register first! I won’t be able to attend, but my colleagues John Brandenburg and Kalpana Goel will be there mentoring. Go sign up now!

But what about authenticated users? You can cache elements of the page using a method like Render cache, Entity Cache, or Views Content Cache. But Drupal still has to assemble each page for your users, a relatively heavy operation! If you want to address hundreds or thousands of authenticated users, you’re simply SOL by these traditional approaches.

Enter the Auth Cache suite of modules. Though this project has been around for quite some time, it had a reputation of being finicky and hard to set up. It got a significant rewrite in the last year thanks to znerol, and is now a powerhouse of a module that brings authenticated user caching much closer to regular users.

I will say that this is still not for the squeamish. You have to really understand the building blocks of your site, and you will have to make a plan for each unique layout on your site. There are some page elements that are quite hard to build this way, but for the most part Authcache makes this easy.

The theory

The idea behind authenticated user caching is simple. We already have a great caching mechanism for pages that stay exactly the same for all users. So we simply identify the parts of the page that will change for each user, and use a placeholder for them instead. Think of it as a tag in HTML. This way the page caching mechanism can ignore the customized content, and focus on the stuff that IS the same across all requests.

There are three major ways of doing this placeholder: AJAX, ESI, and Cookies.

With AJAX, you just include a little JS that says “fill this DIV with the contents of http://example.com/user/customized/thing”. The client’s web browser makes a second call to the server, which is configured to allow /user/customized/thing through the cache all the way to your website. Drupal (or whatever you’re running) fills in the HTML that belongs in that div and returns it to the browser. Congratulations! You just served an authenticated user a page which was 99% cached. You only had to generate the contents of one div.

ESI is short for Edge Side Includes, a small extension to HTML which effectively does the same thing as that Javascript, but on the “Edge server”. The Edge server is whatever service touches the HTTP request last before sending it to the client. Apache, NGINX, Varnish, pound… you want this to happen as far down the stack as you control. An ESI tag in your HTML looks like this:

1

<esi:include src="http://example.com/user/customized/thing" onerror="continue"/>

It’s pretty clear, even to a human reader, what this tag means: “replace this tag with the contents of http://example.com/user/customized/thing”. ESI actually supports some simple logic as well, but that’s not really relevant to what we’re doing here.

The only difference between ESI and AJAX is where the placeholder is filled. With ESI it happens on the edge service, and with AJAX it happens in the client browser. There is a subtle difference here: a page with ESI will not be delivered until all the ESI calls have returned something, while an AJAX page will return right away, even if the components don’t immediately appear. On the other hand, ESI is much better for degraded browsers. YMMV.

The last method is using Cookies. You can store arbitrary information on cookies, as long as you’re careful about security. That can be a very effective way to get certain limited information through a caching layer. Authcache actually comes with an example module for just such a use case. It passes the user’s name and account URL in a cookie, so you can display it in a block.

This is effective for very small amounts of information, but keep it limited. Cookie headers aren’t designed to hold large quantities of data, and reverse proxies can have a hard time if you put too much information in there. Still, it’s a neat trick that can cover you for that “Hello Username” block.

Authcache in Drupal

The Authcache suite of modules tries to automatically implement AJAX and/or ESI for you. It actually goes one step further, and implements a caching layer for those “fragments” which are to be returned via ESI/AJAX. The fragments can be stored in any caching system which implements DrupalCacheInterface, ie any caching module you’ve heard of. Memcache, APC, File Cache, Redis, MongoDB. The full page HTML with placeholders can be cached in Drupal’s normal page cache, in Boost, or in Varnish.

Once you have these caching mechanisms defined, it’s just a question of marking sections of your site which need a second callback. Authcache presents a large number of modules to do this. You can define any of the following as requiring a second call:

• Blocks
• Views
• Panels Panes
• Fields
• Comments
• Flags
• Forms
• Forums
• Polls
• Votes

… and that’s all without writing a single line of custom code! Each one of those elements gets a new “Authcache” setting, where you can define it as needing a second callback, and set the method for the callback as either AJAX or ESI. You can even fall back to another method if the first one fails!

A good example of how this works is the Forms integration. Authcache will modify any and all forms on your site, so that they have an ESI or AJAX placeholder for the form token. This means that the form itself can be stored in your page cache (Varnish, Boost, or whatever), and the form token will be filled in automatically! That’s a phenomenal speed improvement without any configuration beyond enabling the module.

Setting up Authcache is a little complicated, and I’ll cover that in depth in my next post. But once the basic AJAX or ESI support is set up and these modules are enabled, caching authenticated users becomes a question of visiting each unique layout on your site and making a plan for each element that involves user customization. Authcache makes this easy.

Next post: How to configure Authcache on Drupal 7.

The evil Lord Over Engineering is threatening to delay the release of the CMS, which we need to save the world! The only way to stop him is to assemble the greatest force of Drupal superheroes ever assembled! Can the heroes save the day? Can we manage to make the final git push? You’ll have to be there to find out!

“If you only get up early once during DrupalCon, this is the morning to do it. And hey, at least you’ll get better seats for my keynote right after.” — Dries

In Prague we had the Drupal Opera, with solos sung by Gabor Hojtsy. In Portland we had the Drupal Game show, including Doug Vann’s amazing beatbox of the Tetris theme. In Munich, we taught the world to yodel and pour good German beer. Don’t miss out this year! The fun is just getting started!

If you want to participate onstage, you can go to Robert Douglass’ blog and sign up with our superhero/villain application form. But even if you just want to party from your comfy chair in the audience, costumes are encouraged! So get your best superhero costume together, and I’ll see you at the pre-keynote!

Lauri and I met at the last Drupal Dev Days event, in Szeged. That was also hailed as an example of a hugely successful Drupal event, and he took the lessons from their in-depth report to heart. To be fair, the local volunteers and sponsors also clearly busted their humps getting people registered, and finding good session speakers to work with.

The result was a really positive Drupal event for all of us. Their attendance shot past the 200 mark for the first time, their code sprint had more involvement than ever before, and their social activities were a huge success. We left Finland full of positive feeling for the local association there, the city of Helsinki, and of course the sauna culture! This was a great example of what a Drupal community event can be. I’m looking forward to next year already.

If you didn’t catch it at Drupal Dev Days in Szeged, or at Drupalcamp Frankfurt, you’re probably going to have to wait for Drupalcon Amsterdam to take part! But I do have a video of the session at Frankfurt, just to whet your appetite. :)

You can consider this a challenge: if any other themers out there want to challenge me to a coder vs themer style battle, I’ll be keynoting at Drupalcamp Helsinki in a few weeks. I’ll meet you there!

But just as in Panels you sometimes need a pane that isn’t provided out of the box, in Display Suite you sometimes want to add a field that isn’t really a field on your content. In a recent site build, I used this capability to include information from the Organic Groups a user belongs to on his profile as it appears in search results.

DS offers some ability to create this kind of custom field through the UI, but I’m talking about more complicated outcomes where you need/want to use custom code instead. This is actually even easier than custom panels panes.

In our example, we will display the user’s name, but backwards. Obviously you can do much more complex things with this, but it’s nice to have a simple example!

Declare your fields

First we have to tell Display Suite about our new custom field. We do this with hook_ds_fields_info().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

<?php

//@file: Add a custom suite to display suite for Users.

/**
 * Implements hook_ds_fields_info().
 * Declare my custom field.
 */
function mymodule_ds_fields_info($entity_type) {
  $fields = array();

  if ($entity_type == 'user') {
    $fields['backwards_username'] = array(
      'title' => t('Backwards Username'),
      'field_type' => DS_FIELD_TYPE_FUNCTION,
      'function' => 'mymodule_backwards_username',
    );
  return array($entity_type => $fields);
  }
  return;
}

Any guesses whathappens next? That’s right, we have to write our render function under the name we just declared. You can put anything here, really anything renderable at all.

1
2
3
4
5
6
7
8

/**
 * Render function for the Backwards Username field.
 */
function mymodule_backwards_username($field) {
  if (isset($field['entity']->name)) {
    return check_plain(strrev($field['entity']->name));
  }
}

That’s it. So simple, you’ll wonder why you ever did it any other way!

One of Drupal’s greatest strengths is the closness of its’ community, how friendly and accepting they can be. Drupalcons are highlight events for many, not because of the learning as much as because of the social track: the chance to see old friends and make new ones. Even more important is the chance to experience in person this incredibly friendly community. I always loved the cons because you could approach really anybody, say “hi”, and ask them about their work with the platform. Seriously, anybody. From a new user to Dries himself.

That’s become harder and harder as Drupal has grown more popular. In a convention of more than 3,000 people, you lose that feeling of being able to approach anybody. Instead, people silo into groups. In a best case it’s a group that shares an interest in a sub-system (Rules junkies, Panels proselytizers, Features fans…), but in most cases it’s because of shared connections outside the community. You end up hanging out with the same people you knew before the con. Of course you can still have fun, but that sense of community is lost.

One of the best parts of Drupal Dev Days Szeged was the way they encouraged people to mix, cross pollinate, and discuss. In a conference of 350 people I felt like I spoke to almost all of them. I could approach even the famous visitors and talk to them like a normal human being. I borrowed VGA adaptors from Gabor Hojtsy and Wim Leers, and neither of them batted an eye at it.

This kind of experience is so great, so positive and validating, that I recommend Drupal Camps for everyone. The ticket price is cheap, the location is always nearby, and the culture is fantastic. The sessions are every bit as good as most DrupalCon sessions (many of us use the Camps as a way to practice before the Con), and you will make great new friends.

Tl;DR: Drupal Dev Days in Szeged was fantastic. If you’ve never been to a Drupal Camp event, get your butt onto drupical.com and find your nearest one today!

And I’ve also ignored the warning message every time before, but today I thought I’d check it out:

WARNING: Using temporary files to store and transfer sql-dump. It is recommended that you specify —source-dump and —target-dump options on the command line, or set ‘%dump’ or ‘%dump-dir’ in the path-aliases section of your site alias records. This facilitates fast file transfer via rsync.

There are actually two possible solutions to this warning (that I can think of), and they illustrate some of the useful “power user” features of Drush that any frequent user should be aware of.

The warning is there because drush would prefer to rsync the DB dump from site1 to site2, rather than a one time copy. Rsync has lots of speed improvements, not the least being diff transfer. When transferring an updated copy of a file which already exists at the destination, rsync will only send over the changes rather than the whole file. This is pretty useful if you’re dealing with a large, text based file like an SQL dump – especially one that you’ll be transferring often. In order to use this efficient processing though, Drush needs to know a safe path where it can store the DB dump in each location.

First we’ll add the %dump-dir% attribute to our alias for clientsite:

1
2
3
4
5
6
7
8
9
10
11
12
13

<?php
// Site clientsite, environment live 
$aliases['live'] = array(
  'parent' => '@parent',
  'site' => 'clientsite',
  'env' => 'live',
  'root' => '/var/www/example.com/public_html',
  'remote-host' => 'example.com',
  'remote-user' => 'cvertesi',
  'path-aliases' => array(
    '%dump-dir' => '/home/cvertesi/.drush/db_dumps',
  ),
);

Notice that %dump-dir actually goes in a special sub-array for path-aliases. This is very likely the only time you’ll need to use that section, since most everything else in there is auto-detected. This is the directory on the remote side where drush will store the dump.

Our options come in with the @self alias. In a local dev environment, the most common way to handle this is in your drushrc.php file:

1

$options['dump-dir'] = '~/.drush/db_dumps';

But this won’t work for all cases. You can also take advantage of Drush’s alias handling by creating a site alias with the settings you want, and letting Drush merge those settings into @self. When Drush builds its’ cache of path aliases, it uses the site path as the cache key (for local sites only). That means that if you have a local alias with the same path as whatever @self happens to resolve to, your alias options will make it into the definition for @self. So here’s the alternate solution:

1
2
3
4
5
6
7

$aliases['localdev'] = array(
  'root' => '/Users/cvertesi/Sites/clientsite',
  'uri' => 'default',
  'path-aliases' => array(
    '%dump-dir' => '/home/cvertesi/.drush/db_dumps',
  ),
);

There’s just one, obscure caveat with the latter method: somewhere in the alias merging process, BASH aliases are lost. That means that ‘~’ stops resolving to your home directory, and you have to write it out (as I did above).

Have fun!

Very often your site will diverge from the install profile as it takes on a life of its own, and it will be useful to convert it to “vanilla” Drupal. Today I’ll use a relatively simple example of a musician site which is moving away from the Pushtape distribution. Later I’ll return to this subject with the much more in-depth example of moving a community site away from Drupal Commons.

Move things around

Install profiles have all their files stored in the site root’s profiles/ directory. The first step is going to be moving everything out of there. In the case of pushtape, we have libraries, modules, and a theme stored in there. We’re going to move them to a more normal location.

1
2
3
4
5
6
7
8
9

# mkdir sites/all/libraries
# mv profiles/pushtape/libraries/* sites/all/libraries

# mkdir sites/all/modules/custom
# mv profiles/pushtape/modules/pushtape_* sites/all/modules/custom
# mv profiles/pushtape/modules/* sites/all/modules

# mkdir sites/all/themes
# mv profiles/pushtape/themes/* sites/all/themes

Next we need to see if there are any variables set in the install profile which really depend on the profile directory. If there are, we’ll have to set them again with the new path.

1
2
3

# cd profiles/pushtape
# grep 'profiles/pushtape' * -R
pushtape.install:  variable_set('sm2_path', 'profiles/pushtape/libraries/soundmanager2');

In this case, we see one variable_set which tells the system where to find the soundmanager2 library. We can update that easily enough with drush:

1

# drush vset sm2_path 'sites/all/libraries/soundmanager2'

Now we have to update Drupal’s setting for which install profile was used to create the site.

1

# drush vset install_profile standard

In some cases this will be enough to work. Personally I like to keep my modules folder more organized, so I go the extra mile:

1
2
3

# cd sites/all/modules
# mkdir contrib
# mv !(custom|contrib) contrib

I also separated out the custom code from the features. You can figure out which custom modules implement features with find . |grep features, and move them into a separate directory manually.

Clearing caches

Once you’re done moving things around, CLEAR CACHES. Drupal keeps an index of module, library, and theme directories, and you just broke it.

1

drush cc all

The only problem is, in many cases you will have moved a module that is required for drupal bootstrap. So you’ll have to get the handy drush tool Registry Rebuild, and run that before your cache clear:

1
2
3

# drush dl registry_rebuild
# drush rr
# drush cc all

Extra Cleanup

As commenter @ericaitala notes, you may need some followup cleanup to really get all traces out. The easiest way to do this is from the SQL command line, which you can access via drush:

1
2

drush sqlq "DELETE FROM `system` WHERE filename LIKE 'profiles/profilename/profilename.profile"
drush sqlq "UPDATE `system` SET status=1 WHERE filename LIKE 'profiles/standard/standard.profile'"

Technically these should both be covered by the registry_rebuild operation, but we’re doing it by hand because it seems to be missed in some operations. The first command removes the entry for the profile from Drupal’s system table – it removes any knowledge Drupal has that there was an install profile there. The second tells Drupal that the “standard” install profile is active, and should be checked for updates.

That’s it – your site is now officially a vanilla Drupal install. Test by removing the profiles/pushtape directory, clearing caches, and browsing around your site.

NOTE: With a more complex install profile I expect to encounter more difficulty. Stay tuned for the post on extricating yourself from Commons later this year!

As usual, I’ll assume that you have an empty custom module called mymodule, with only a .info and a .module file to its name.

1) Tell CTools that you have custom code here

Ctools, like Views, needs a hook to declare the fact that you have custom code. To do this we’ll use hook_ctools_plugin_directory. This hook is invoked for all Ctools plugin types, and includes the module name as a variable. This way you can avoid eating up memory for anything except the targeted module. You also have to declare where your custom code will live. So here’s the complete content of mymodule.module:

1
2
3
4
5
6
7
8
9
10

<?php

/**
 * Implements hook_ctools_plugin_directory().
 */
function mymodule_ctools_plugin_directory($owner, $plugin_type) {
  if ($owner == 'ctools' && $plugin_type == 'content_types') {
    return 'plugins/content_types';
  }
}

Note: Do not confuse Ctools “Content Types” with the “Content Type” entity used elsewhere in Drupal. This is just confusing naming, but they’re totally different things. Actually the most common usage for a Ctools Content Type is a pane, just like what we’re doing now. There are other plugin types, but none that interest us in this post.

2) Add your custom pane

Oh, did you think this would be more difficult? Now that we’ve told Ctools to look for Content Type plugins in our module’s plugins/content_types subdirectory, we just add a .inc file for each “Content Type” (aka Pane) that we want to add. Let’s do a simple one, which returns the root term of a given taxonomy term. All the following code will go in plugins/content_types/taxonomy_root_term.inc (a name I chose arbitrarily).

Right at the top of the file, we provide a $plugin array which defines the basic information about our Pane Ctools Content Type. This doesn’t go into a function or anything, it just sits at the top of the .inc file:

1
2
3
4
5
6
7
8
9
10
11
12
13

<?php

$plugin = array(
  'single' => TRUE,
  'title' => t('Taxonomy root term'),
  'description' => t('a Display of data from the root term of the given TID'),
  'category' => t('Custom Panes'),
  'edit form' => 'mymodule_taxonomy_root_term',
  'render callback' => 'mymodule_taxonomy_root_term_render',
  'admin info' => 'mymodule_taxonomy_root_term_info',
  'defaults' => array(),
  'all contexts' => TRUE,
);

As you can see, this array defines a category, title, and description for the Panels admin interface. It also declares the names of the callbacks which provide the pane’s edit form, rendered form, and admin info. “Single” means that this type has no sub-types. This is the case in every single custom pane I’ve ever seen, so it’s probably the case for yours as well.

Now we write the callbacks we named in that array. We’ll start with the edit form.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

/**
 * Edit form.
 */
function mymodule_taxonomy_root_term($form, &$form_state) {
 $conf = $form_state['conf'];

 $form['term'] = array(
   '#type' => 'textfield',
   '#title' => t('Term ID'),
   '#description' => t('The term, from which the root term will be displayed'),
   '#default_value' => $conf['term'],
 );

  $entity_info = entity_get_info('taxonomy_term');

  $options = array();
  if (!empty($entity_info['view modes'])) {
    foreach ($entity_info['view modes'] as $mode => $settings) {
      $options[$mode] = $settings['label'];
    }
  }

 $form['view_mode'] = array(
   '#type' => 'select',
   '#options' => $options,
   '#title' => t('View mode'),
   '#default_value' => $conf['view_mode'],
 );

 return $form;
}

This is a fairly standard Drupal form. It also goes through typical form validation and submission functions, so you can provide a pretty complete experience for the administrator. In our case, we just want to get the term ID of the term whose root parent should be displayed. We let the administrator enter the term ID, and the view mode which should be used to display it. We won’t worry about form validation in our example. Let’s move on to the Submit function:

1
2
3
4
5
6
7

/**
 * Edit form submit function.
 */
function mymodule_taxonomy_root_term_submit($form, &$form_state) {
  $form_state['conf']['term'] = $form_state['values']['term'];
  $form_state['conf']['view_mode'] = $form_state['values']['view_mode'];
}

Again, pretty simple stuff. We just make sure that the $form_state[‘conf’] has the values entered. Now, the next callback we defined in $plugin is for rendering the pane:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

/**
 * Render the panel.
 */
function mymodule_taxonomy_root_term_render($subtype, $conf, $args, $contexts) {
  if ($context->empty) {
    return;
  }
  // Get full term object for the root term.
  $term = ctools_context_keyword_substitute($conf['term'], array(), $contexts);
  $parent_array = taxonomy_get_parents_all($term);
  $root = end($parent_array);

  // Render as a block.
  $block = new stdClass();
  $block->module = 'entity';
  $block->delta = 'taxonomy_term-' . str_replace('-', '_', $conf['view_mode']);

  $entity = entity_load_single('taxonomy_term', $root->tid);
  $block->content = entity_view('taxonomy_term', array($root), $conf['view_mode']);
  return $block;

}

First we make sure there is information – ie the taxonomy term ID we need – in the pane’s context. Then we get the root term object and render it in the requested display mode. The only requirement for the return here is that it be a Drupal render array. So depending on your use case, you can return an image, a field… whatever you like. In most cases a block is a convenient wrapper for whatever you have to return, which is what I did here.

This is as far as you have to go. The admin info callback isn’t actually required, just don’t include it in the $plugin array and you’ll be fine. But if you want to make your life easier as a site admin, it’s definitely a nice to have.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

/**
 * Admin info.
 */
function mymodule_taxonomy_root_term_info($subtype, $conf, $contexts) {
  if (!empty($conf)) {
    $content = '<p><b>Term ID:</b> ' . $conf['term'] . '</p>';
    $content = '<p><b>View mode:</b> ' . $conf['view_mode'] . '</p>';

    $block = new stdClass;
    $block->title = $conf['override_title'] ? $conf['override_title_text'] : '';
    $block->content = $content;
    return $block;
  }
}

This just provides the administrative summary which you can see in the Panels UI. Again, Panels will be happy with any render array return you throw at it, so I use a block.

This is why we have nice things

Anyone who’s worked with me knows that I’m not a huge fan of the panels everywhere approach. But I use it often, simply because it makes custom layouts and totally custom page pieces so easy to do. Duplicating even this very simple functionality in a block is actually harder than this. You’re still using about 3 functions, but you’d have to determine in advance where that TID will come from. It would certainly come out less flexible, and actually probably harder to maintain. With Ctools all your related code sits in one place, and your module structure actually helps you see what’s going on where.

If you learn how to do elements like this, you’ll find Panels creeping into more and more of your builds. And rightfully so.

“…it is highly advisable to use regular handlers and plugins when available (or even to create one yourself). Take note that filtering and sorting a view using PHP always has a considerable perfomance impact.”

As of this writing, 44,497 site maintainers have read that warning and chosen to ignore it. They’ve chosen to put their PHP into a non-revisioned, difficult-to-access place, and to enable PHP input in a module that was never designed for security. They’ve left their site at risk of a very difficult to diagnose and even harder to fix WSOD.

I’m going to go out on a limb here, and suggest that in many of these cases, the decision was made because someone had the impression that writing a Views handler or Plugin was difficult. I’m here to tell you that’s not so: it’s actually quite easy.

What we’re doing

We’re going to tell Views about the structure of the data we want to display, filter, or sort – even if there’s not actually a new data source involved, that’s how you do it – and then we’ll write the function that actually does the filter/sort/etc by improving an existing field display/filter/sort that Views already includes.

This process will work for:

• Defining a new data source for Views, ie something your module keeps in the DB.
• Creating multiple field displays/filters/sorts for an existing field.
• Creating a completely computed field display/filter/sort, with nothing in the DB.

I know that in 99% of use cases for Views PHP, you don’t need to define a new data source, table, adn fields. Trust me that this is the easiest way to learn it, though. I promise we’ll get to your use case before the end of the post.

How to

I’ll assume you have a custom module built, with a .info and .module file, but nothing in there yet. We’ll call our module “mymodule” for the example.

1) Tell Views about your module

We implement hook_views_api to let Views know that our module provides some code for Views, and what version of the Views API we’re using.

1
2
3
4
5
6
7
8
9
10
11

<?php

/**
 * Implements hook_views_api().
 */
function mymodule_views_api() {
  return array(
    'api' => 3,
    'path' => drupal_get_path('module', 'mymodule') . '/views',
  );
}

Couldn’t be simpler. We declare that we’re using Views API 3, and that our Views code will all live in the /views subdirectory of our module.

2) Tell Views about your custom code

Now that Views knows to look in our /views directory, we should populate it. Views will look for a file called modulename.views.inc in that directory, so this is where we will put our Views hooks. There are lots of Views interventions you can do in this file, but we’re only interested in one: hook_views_data.

This hook lets you define new data sources to Views, and for each one show how to render a field, how to Filter results, and how to Sort results based on your new data source. I promised you three use cases up there though, and here’s the trick: you don’t have to have an actual data source. You can define a filter for a database field that’s already described elsewhere.

First let’s look at a real field definiton though, because it’s simpler. Here’s how we would define a real DB table as a data source. The table looks like this:

So here’s our implementation of hook_views_data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

/**
 * Implements hook_views_data().
 */
function mymodule_views_data() {
  // Build an array named after each DB table you're describing. In our case,
  // just mymodule_table.
  $data['mymodule_table'] = array(
    // First give some general information about the table as a data source.
    'table' => array(
      // The grouping for this field/filter/sort in the Views UI.
      'group' => t('Example Views Stuff'),
      'base' => array(
        'field' => 'naid', // This is the identifier field for the view.
        'title' => t('Example Views API Data'),
        'help' => t('Names provided by the Mymodule module.'),
      ),
    ),
    // Now we describe each field that Views needs to know about, starting 
    // with the identifier field.
    'naid' => array(
      'title' => t('Name ID'),
      'help' => t("The unique Name ID."),
      'field' => array(
        'handler' => 'views_handler_field_numeric',
        'click sortable' => TRUE,
      ),
      'sort' => array(
        'handler' => 'views_handler_sort',
      ),
      'filter' => array(
        'handler' => 'views_handler_filter_numeric',
      ),
    ),
    // Now the name field.
    'name' => array(
      'title' => t('Name'),
      'help' => t("The Name."),
      'field' => array(
        'handler' => 'views_handler_field',
        'click sortable' => TRUE,
      ),
      'sort' => array(
        'handler' => 'views_handler_sort',
      ),
      'filter' => array(
        'handler' => 'views_handler_filter_string',
      ),
    ),
  );
  return $data;
}

This is a pretty simple example, and I think the array structure speaks for itself. First you provide some general information about the table, then you create a sub-array for each field on the table. Each field’s array should be named after the field, and provide at least title. Of course it wouldn’t be useful if you didn’t describe the handlers for any field/sort/filter operations you want to expose. For each one of these you just provide the name of the handler. In this example I used all built-in filters that come with Views, but it’s easy enough to provide a custom handler.

Many added behaviors in Views start with hook_views_data; this only covers the basics. You can also open fields up as arguments or relationships, or even add built-in relationships. For example, if our table also contained an NID field, we could define a relationship so that node fields are always available when listing names, and vice versa. This stuff is all surprisingly easy to do, it’s just not the focus of this post.

3) Write your custom handler

Let’s say we want to provide our own field handler for the name field. Maybe we want it to automatically separate first names. This is easy, too! You simply decide on a name for your new handler – by convention it should begin with modulename_handler_type_, so we’ll use mymodule_handler_field_firstname. Here’s the relevant part of that $data array from before:

1
2
3
4
5
6
7
8
9
10

...
    // Now the name field.
    'name' => array(
      'title' => t('Name'),
      'help' => t("The Name."),
      'field' => array(
        'handler' => 'mymodule_handler_field_firstname',
        'click sortable' => TRUE,
      ),
...

Not exactly rocket science, is it?

Now we create a file named after the handler, also in the /views subdirectory. Though you could write your own handler class from scratch, you’ll almost never have to. It’s much easier to just extend an existing class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php

/**
 * @file
 * Definition of mymodule_handler_field_firstname.
 */

/**
 * Provide the first name only from the name field.
 *
 * @ingroup views_filter_handlers
 */
class mymodule_handler_field_firstname extends views_handler_field {
  /**
  * Render the name field.
  */
  public function render($values) {
    $value = $this->get_value($values);
    $return = explode(' ', $value);
    return 'First name: ' . $return['0'];
  }
}

You see the pattern we’re following: just name a handler, then extend an existing Views handler class to do what you want. You can override options forms, the admin summary… really any aspect of the way Views handles this data. And the pattern is the same for fields, sorts, filters, and arguments.

Once you’ve created your handler’s .inc file, you have to make sure your module loads it. So edit your module’s .info file thusly:

1
2
3
4
5

name = My Module
description = Demo module from ohthehugemanatee.org
core = 7.x

files[] = views/mymodule_handler_field_firstname.inc

4) Multiple filters for one field

We all understand how this works for data that you’re declaring for the first time in Views. But what if you want to provide multiple handlers for a single field? Maybe there are several different ways to filter or sort it. For most use cases, you should just follow the pattern above, and simply override the Views options form in your handler class. But occasionally you really do need multiple handlers.

So let’s add a second and third field handler for our name field:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

...
    // Now the name field. This is the first, and 'real' definition for this field.
    'name' => array(
      'title' => t('Name'),
      'help' => t("The Name."),
      'field' => array(
        'handler' => 'mymodule_handler_field_firstname',
        'click sortable' => TRUE,
      ),
    ),
    'name_last' => array(
      'title' => t('Last name'),
      'help' => t('The Last name, extracted from the Name field'),
      'real field' => 'name',
      'field' => array(
        'handler' => 'mymodule_handler_field_lastname',
        'click sortable' => TRUE,
      ),
    ),
    'name_backwards' => array(
      'title' => t('Evil Genius Name'),
      'help' => t('The name, reversed so it sounds like the name of an evil genius.'),
      'real field' => 'name',
      'field' => array(
        'handler' => 'mymodule_handler_field_evil',
        'click sortable' => TRUE,
        ),
      ),
...

Can you spot the difference? All you have to do is add a variable for real field, which tells Views the field name to use for the source value, and that’s it. Everything else is totally identical to a normal field. By custom we prefix the “virtual” field’s name with the name of the real field, but that’s as complicated as it gets.

Conclusion

If there’s one thing I want you to take away from this blog post, it’s that the Views API is actually really easy. And if you can’t find something online, take a moment to actually look at the API documentation included with the module. It’s very thorough, and easy to read. If you feel like you understand how this works, but the doco doesn’t quite cover what you’re trying to do, look for examples in the Views module itself! There are 169 handlers for every concievable kind of case, just within Views. Find something reasonable and build off of that!

With this in mind, it’s only 24 lines of simple code to provide your own handler for an existing field. After that 24 lines, you’re doing the same things you were planning to do in views_php… but now you’re doing them in a real coding environment, with a revisioning system, and where it’s easy to track down and fix errors that could otherwise crash your site. 24 lines of array definition can save you a world of hurt. I hope to see those views_php installation numbers dropping soon.

My task was to determine if the displayed node was entity-referenced as being the “special” node from it’s parent organic group. It’s a weird requirement (which is exactly why a custom Condition makes sense here), so let me explain that again. On a site with Organic Groups, the Group node has an entityreference field, which marks one of the Group member nodes as special. When the user is viewing this special node, our Rules condition should evaluate to positive.

The first prerequisite is to make absolutely certain that you can’t do this using any of the built in Conditions, and something this unique definitely qualifies there. So let’s get to the implementation in our custom module. The module will be called CCC for Custom Context Condition.

1
2
3
4
5
6
7
8
9

name = CCC (Custom Context Condition) Example Module
description = Provides a custom Context Condition
core = 7.x
version = 7.x-1.x
package = Custom

dependencies[] = entityreference
dependencies[] = context
dependencies[] = og

That’s a totally normal .info file, with logical dependencies on OG, EntityReference, and Context modules. Let’s have a look at the .module file. This is probably a lot simpler than you expected.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

/**
 * Impelements hook_context_plugins().
 */
function ccc_context_plugins() {
  $plugins = array(
    'ccc_condition_og_special_node' => array(
      'handler' => array(
        'path' => drupal_get_path('module', 'ccc') . '/plugins/context',
        'file' => 'ccc_condition_og_special_node.inc',
        'class' => 'ccc_condition_og_special_node',
        'parent' => 'context_condition',
      ),
    ),
  );
  return $plugins;
}

First we implement hook_context_plugins(), to declare our new condition plugin to Context. This function should return an array of plugins, keyed by plugin name (in our case, ccc_condition_og_special_node). For each plugin, you have to explain to Context some basic information about the handler you’re going to write.

• path The path to the plugin file. By convention you should put it in your module’s directory, under /plugins/context.
• file The filename to look for. Keep yourself sane, and name it after the plugin you’re writing.
• class The name of the Class you’re about to write. If you’ve never written a PHP class before, this is good practice for D8 and object oriented code in general. Think of it like a function name, and again: name it after the plugin you’re writing.
• parent The Class you are extending to create your condition. If you don’t know what to put here, just enter ‘context_condition’.

Now that Context knows about your plugin, you have to declare it to the UI in order to use it! For this we implement hook_context_registry. This function returns an array keyed by plugin type—in this case, “conditions”. For each condition (keyed by condition name), we need title, description, and plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

/**
 * Implements hook_context_registry().
 */
function ccc_context_registry() {
  $registry = array(
    'conditions' => array(
      'ccc_condition_og_special_node' => array(
        'title' => t('OG Special Node'),
        'description' => t('Set this context based on whether or not the node is the "Special Node" entityreferenced in the parent OG.'),
        'plugin' => 'ccc_condition_og_special_node',
      ),
    ),
  );
  return $registry;
}

Now Context module knows everything it needs to know about your plugin and condition, we have to tell Drupal when to evaluate your condition. You can implement whatever hook make sense for you here, the important part is that you execute your plugin. Since our condition only makes sense after everything else has fired (ie when the OG context is well and firmly set), we’ll implement hook_context_page_reaction().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

/**
 * Implements hook_context_page_reaction().
 *
 * Executes our OG Special Node Context Condition. 
 * Gotta run on context_page_reaction, so Views and OG have a chance to
 * set/modify Group context. 
 */
function ccc_context_page_reaction() {
  $group = og_context();
  // Only execute the group node context condition if there is a group node
  // in context.
  if ($group) {
    $plugin = context_get_plugin('condition', 'ccc_condition_og_special_node');
    if ($plugin) {
      $plugin->execute($group);
    }
  }
}

That’s it for your module file. Just declare the plugin to Context and its UI, and find a place to actually execute the plugin. Now we’ll write the actual handler class.

Create your plugin file in the place you promised Context to find it in your hook_context_plugins() implementation. In our case, this is plugins/context/ccc_condition_og_special_node.inc . We’re going to extend Context’s basic Condition Class to provide our own functionality. Here are the contents of my ccc_condition_og_special_node.inc file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

<?php
/**
 * Expose Web Area Contact Form as a Context condition.
 */
class ccc_condition_og_special_node extends context_condition {
  function condition_values() {
    return array(
      'TRUE' => t('Node is an OG special node'),
      'FALSE' => t('Node is not an OG special node'),
    );
  }

  function condition_form($context) {
    $form = parent::condition_form($context);

    $form['#type'] = 'radios';
    if(empty($form['#default_value'])){
      $form['#default_value'] = 'TRUE';
    }
    else{
      $form['#default_value'] = current($form['#default_value']);
    }
    return $form;
  }

  /**
   * Condition form submit handler.
   *
   * Storing values in an array since that's what Context prefers
   */
  function condition_form_submit($values) {
    return array_filter(array($values => $values));
  }

  function execute($group) {
    if (!empty($group) && $group['group_type'] == 'node') {
      $group_node = entity_load_single($group['group_type'], $group['gid']);
      $group_wrapper = entity_metadata_wrapper('node', $group_node);
      $special_nid = $group_wrapper->field_special_node->value();

      foreach ($this->get_contexts() as $context) {
        $values= $this->fetch_from_context($context, 'values');
        if (arg(0) == 'node' && arg(1) == $contact_nid->nid && !empty($values['TRUE'])) {
          // This is a special node, and the condition is set to match special 
          // nodes.
          $this->condition_met($context);
        }
        if (arg(1) != $contact_nid->nid && !empty($values['FALSE'])) {
          // This is not a special node, and the condition is set to match
          // non-special nodes.
          $this->condition_met($context);
        }
      }
    }
  }
}

The trickiest part of this is in the Condition settings form and values. Context assumes that your settings form will be a series of checkboxes, and does a lot of internal processing based on that assumption. We don’t want to mess any of that up, so there’s a bit of dancing around the requirement here.

First we provide the function condition_values. Context needs to know in advance what the possible values are for the Condition’s settings form, and this is where you return them. Based on this return, Context will build a settings form of checkboxes for you.

Then we override the settings form with condition_form(). I change the type of the form element to radio boxes, and set a default value.

Then I add my own submit handler, which merely takes the result of the radio box and puts it into an array, just like it would be if this was a checkbox.

Finally, we get to the good part: the execute function. If you recall, this is what we called in ccc_content_page_reaction(). Here we load the Group node, and use entity_metadata_wrapper to extract the value of the field_special_node entityreference field on that node. Then we test the current NID from the URL. Note that you never have to explicitly return FALSE; Context is only watching for TRUE returns.

When I learned how to do this, I found it surprisingly easy. The hardest part is wrestling with the Condition class to get exactly the behavior you like. Everyone ends up doing some dancing around here, so don’t feel bad about it. Context’s own Conditions are great examples. Have a look at the classes provided in context/plugins/context_condition_*.inc to get ideas for how to do this.

What’s fantastic about Rules is that it lets everyone who wants your module to behave slightly differently to change the behavior for themselves. In fact, if you build the logic of your module into Rules you get an excuse to reject half the tickets in your contrib module queue on the grounds of “works as intended,” aka “do it yourself.”  To me, this is the quiet brilliance of Commerce module’s dependence on Rules. Their code is almost 100% functional elements, with as little behavioral logic as possible. All of the logic is written in Rules. Of course they still distribute Commerce with a hefty set of default rules, so it works with a standard implementation out of the box. But at the same time, it is open to completely custom workflows without the developers ever having to get involved.

Today we’re going to cover adding Rules to existing code. This is an easy way to write solid and useful patches for other contrib modules out there, patches that get used. For your own projects you will probably want to implement this code in your own custom module, just hooking into core or contrib behaviors. Either way the logic is the same.

At it’s core, the Rules API lets you expose functions you’ve already written to the GUI. 90% of the work is just declaring the functions. There are three elements that Rules cares about:

• Events: Potential triggers that a user can use to take action with Rules.
• Actions: Functions which take a set of parameters and do something. 
• Conditions: Test functions, which return TRUE or FALSE.

You declare these elements to Rules with hooks, usually assembled into a separate modulename.rules.inc file. First, let’s define an event with hook_rules_event_info().

<?php
/**
* Implements hook_rules_event_info().
* Fire a Rules event when a node is validated.
*/
function mymodule_rules_event_info() {
  return array(
    'mymodule_node_is_being_validated' => array(
      'group' => t('My module'),
      'label' => t('A node is being validated!'),
      'variables' => array(
        'node' => array(
          'type' => 'node',
          'label' => t('unsaved node being validated'),
        ),
        'form_id' => array(
          'type' => 'text',
          'label' => t('Form ID'),
        ),
      ),
    ),
  );
}
?>

This implementation tells rules that you are defining an Event with the machine name mymodule_node_is_being_validated. In the Rules dropdown select box that lets you choose the triggering action for a new rule, your Event is called “A node is being validated!” and is sorted into the group “My Module”.  The variables array sets the variables that are offered with this particular event. Each variable in the array must have a  machine name (the key), a human-readable label and a data type. Now, anywhere that you want to fire this rules event, you call it with rules_invoke_event(‘mymodule_node_is_being_validated’, $node) to fire the Event. In this case, you could hook into node validation and add it easily enough:

<?php
/**
* Implements hook_node_validate().
* Fires our Rules event on node validation.
*/
function mymodule_node_validate($node, $form, &$form_state) {
  rules_invoke_event('mymodule_node_is_being_validated', $node, $form['#form_id']);
}
?>

This example brings up one of the big limitations of Rules: you can’t fail validation. We can set an error message, respond with all the power of Rules, but no matter what you do you cannot make this node form fail validation. The good people at rules_forms module are trying to address this, but it’s a much thornier problem than it looks.

Next, let’s define a Rules action with hook_rules_action_info(). This time we’ll use a real world use case from a recent client: our Rules Action will allow you to subscribe a user to a node with the subscriptions module. Again, we’ll add this info implementation to our mymodule.rules.inc file for clarity.

<?php
/**
* Implements hook_rules_action_info()
* Add a Rules Action for subscribing a user to a node.
*/
function mymodule_rules_action_info() {
  return array(
    'mymodule_subscribe_user_to_a_node' => array(
      'label' => t('Subscribe a user to a node'),
      'group' => t('Subscriptions'),
      'parameter' => array(
        'user' => array(
          'type' => 'user',
          'label' => 'Subscribing user',
        ),
        'node' => array(
          'type' => 'node',
          'label' => 'Target node',
        ),
      ),
    ),
  );
}
?>

You can see that we use the same label, group, and module declarations here. Actions take parameters as well, which are structured very similarly to the variables in events. You don’t have to do anything more to make an action: this is it. When Rules fires the action it will look for a function called mymodule_subscribe_user_to_a_node, and fire it with the parameters provided in the GUI. In this case we wrapped our own function around subscribe module’s functionality, because of the way subscribe’s own subscription function works.

<?php
/**
* Creates a subscription for the user
*
* @param $user
*  The user to subscribe to the node.
* @param $node
*  The node to which the user will subscribe.
*/
function mymodule_subscribe_user_to_a_node($user, $node) {
  module_load_include('inc', 'subscriptions', 'subscriptions.admin');
  if (!is_object($node) || !isset($node->nid)) {
    watchdog('rules_subscriptions', 'Error: object passed is not node. Data: !node', array('!node' => print_r($node, true)), WATCHDOG_ERROR);
    return;
  }
  $form_state = array(
    'values' => array(
      'stype' => 'node',
      'sid' => $node->nid,
      'uid' => $user->uid,
      'author_uid' => NULL,
      'send_interval' => _subscriptions_get_setting('send_interval', $user),
      'updates' => _subscriptions_get_setting('send_updates', $user),
      'comments' => _subscriptions_get_setting('send_comments', $user),
    ),
  );
  drupal_form_submit('subscriptions_add_form', $form_state, 'node', $node->nid);
}
?>

So much for Rules actions - they’re even easier than events!  There is one big defining difference for Actions though: they can provide new variables back to Rules. This is as easy as a provides array in hook_rules_action_info(), and returning an array of values from the actual function called. 

Now let’s look at a Rules condition. No surprise by now, we declare the existence of our Rules Condition through hook_rules_condition_info() in mymodule.rules.inc :

<?php
/**
* Implements hook_rules_condition_info().
* Checks if the given user is admin.
*/
function mymodule_rules_condition_info() {
  return array(
    'mymodule_user_is_admin' => array(
      'group' => t('My Module'),
      'label' => t('User is Admin'),
      'parameter' => array(
        'user' => array(
          'type' => 'user',
          'label' => t('User to test'),
        )
      ),
    ),
  );
}
?>

Once again we see the familiar structure to declare what is about to happen. And just like in an Action, Rules is going to fire a function named after the array key, mymodule_user_is_admin, with the parameters provided. Here we’ll have a very simple function to check this, and it has to return TRUE or FALSE. 

<?php
function mymodule_user_is_admin($user) {
  if ($user->uid == '1') {
    return TRUE;
  }
  return FALSE;
}
?>

Great, so now your module is fully integrated with Rules, right? Well, you have to actually include the Rules logic with the module. Rules provides hook_rules_default_configuration for you to do this, but writing rules by hand is a pain. The shortcut solution is to build your Rules in the Rules GUI and export them, and paste the export into your code. entity_import() is a handy function from the EntityAPI module you can use to translate the export format into the format Rules needs. Here’s a sample (hook_rules_default_configuration must be implemented in mymodule.rules_defaults.inc):

<?php
/**
* Implements hook_default_rules_configuration().
*/
function mymodule_default_rules_configuration() {
  $rules = array();
  $rules['rules_mymodule_behavior'] = entity_import(
    'rules_config', 
    '............');
  return $rules;
}

Note that the new rule name has to be prefixed with rules_ .  That’s all you need to do.

What’s wonderful about these functions is how simple they are. Depending on how your module is written, adding Rules support could be as simple as adding calls to hook_rules_event_info(), hook_rules_action_info(), and hook_rules_condition_info() that connect to existing functions already in your module. In 10 minutes or less, you can open your module up to much more flexibility than was ever possible before. And if you dare to prune the functionality out of your existing code, you can build it in the Rules GUI, export it, and have all your logic exposed and totally customizable in a further 15 minutes. That’s less than half an hour, and you can start training your users to deal with their own problems. :)

Questions? Problems? Leave us a message in the comments…

• EVENT: A Resource Conflict Node Form is Validated: This rule fires during node form validation on Resource Conflict enabled content types. You should use this event if you want to set form errors, or if you want to interact with Rules_forms module. It provides both a node object of the node being created/edited and a form object for use with rules_forms. This is the event trigger for the default Rule.
• CONDITION: Contains a Resource Conflict: Evaluate a node object for conflicts. Returns TRUE if there are conflicts for the node.
• ACTION: Load a List of Conflicting Nodes: Creates a list of nodes that conflict with the given node.
• ACTION: Set a Form Validation Error: Stores a form validation error to be fired the next time a validation hook is called on a conflict-enabled node. This is intended for use with the “A Resource Conflict Node Form is Validated” Event.

drupal_set_message()

dpm()

ddebug_backtrace()

db_queryd()