Pixelite - drupalplanet

10 things I learnt building in Drupal 8

Sean Hamlin — Thu, 19 May 2016 00:00:00 +1200

Introduction

I have had the chance to be involved with 2 fresh builds with Drupal 8 now, I thought I would describe some of the neat things I have found during this time and some of my lessons learned. My hope is that blog post will help you in your journey with Drupal 8.

1. Drupal Console is awesome

Every time you need to generate a custom module, or a new block in a custom module, you can quickly and easily use Drupal Console to produce the code scaffolding for you. This quite easily makes the job of a developer a lot less stressful, and allows you to focus on actually writing code that delivers functionality.

I plucked these example commands that I use frequently from my bash history:

drupal site:mode dev
drupal generate:module
drupal generate:plugin:block
drupal generate:routesubscriber
drupal generate:form:config

Documentation is online but for the most part, the commands are self documenting, if you use the --help option, then you get a great summary on the command, and the other options you can pass in.

The other nice thing is that this is a Symfony Console application, so it should feel very familiar to you if you used another tool written in the same framework.

2. Custom block types are amazing

In Drupal 7 land there was bean which was an attempt to stop making ‘meta’ nodes to fill in content editable parts of complex landing pages. Now, fast forward to Drupal 8, and custom block types are now in Drupal Core.

This basically means as a site builder you now have another really powerful tool at your disposal in order to model content effectively in Drupal 8.

Each custom block type can have it’s own fields, it’s own display settings, and form displays.

Here are the final custom block types on a recent Drupal 8 build:

One downside is that there is no access control per custom block type (just a global permission “administer blocks”), no doubt contrib will step in to fill this hole in the future (does anyone know a module that can help here?). In the mean time there is drupal.org issue on the subject.

I also found it weird that the custom blocks administration section was not directly under the ‘structure’ section of the site, there is another drupal.org issue about normalising this as well. Setting up some default shortcuts really helped me save some time.

3. View modes on all the things

To create custom view modes in Drupal 7 required either a custom module or Dave Reid’s entity_view_mode contrib module. Now this is baked into Drupal 8 core.

View modes on your custom block types takes things to yet another level still as well. This is one more feather in the Drupal site builder’s cap.

4. Twig is the best

In Drupal 7 I always found it weird that you could not unleash a front end developer upon your site and expect to have a pleasant result. In order to be successful the themer would need to know PHP, preprocess hooks, template naming standards, the mystical specific order in which the templates apply and so on. This often meant that a backend and front end developer would need to work together in order to create a good outcome.

With the introduction of Twig, I now feel that theming is back in the hands of the front end developer, and knowledge of PHP is no longer needed in order to override just about any markup that Drupal 8 produces.

Pro tip - use the Drupal Console command drupal site:mode dev to enable Twig development options, and disable Drupal caching. Another positive side effect is that Twig will then render the entire list of templates that you could be using, and which one you actually are using (and where that template is located).

Pro tip: - If you want to use a template per custom block type (to which I did), then you can use this PHP snippet in your theme’s .theme file (taken from drupal.org):

<?php
/**
 * Implements hook_theme_suggestions_HOOK_alter() for form templates.
 *
 * @param array $suggestions
 * @param array $variables
 */
function THEMENAME_theme_suggestions_block_alter(array &$suggestions, array $variables) {
  if (isset($variables['elements']['content']['#block_content'])) {
    array_splice($suggestions, 1, 0, 'block__bundle__' . $variables['elements']['content']['#block_content']->bundle());
  }
}

5. Panelizer + panels IPE is a formidable site building tool

When looking for a layout manager to help build the more complex landing pages, I came across panelizer + panels IPE. Using panelizer you are able to:

create per node layout variants
apply a single layout to all nodes of a particular bundle (e.g. all your news articles have the same layout)

The other neat thing is that the layouts themselves are now standardised between all the various layout managers using a contrib module called layout_plugin. Also they are just YAML and Twig. Simple. There is even an effort to get this merged into Drupal 8.2 which I think would be a great idea.

Downside - all JS is still rendered on the page even though the user (e.g. anonymous users) have no access to panelizer. There is a patch on drupal.org to help fix this.

Since starting this build there has also been a stable release of display suite come out for Drupal 8 as well giving you even more options.

6. You can build a rather complex site with very little contributed modules

For this most recent site I build I got away with using only 10 contributed modules (one of which - devel was purely for debugging purposes).

ctools
google_analytics
metatag
panels
token
contact_block
devel
layout_plugin
panelizer
pathauto

This means you are inherently building a more stable and supportable site, as most of the functionality now comes out of Drupal core.

7. The contact module now is supercharged

In Drupal 7, the contact module was one of those modules to which I never turned on, as it was rather inflexible. You could not change the fields in a UI, nor add email recipients, or have more than 1 form. Now in Drupal 8 you can have as many “contact” forms as you want, each one is fieldable, and can send emails to as many people as needed.

You can also enhance the core module with:

contact_block - allows you to place the contact form in a block
contact_storage - allows you to store the submissions in the database, rather than firing an email and forgetting about it

There is still a place for webform, namely:

large complex form with lots of fields
multi-step forms
forms you want to ‘save draft’

You can read more about this in the OS training blog post on the contact module.

Downside - I wanted to have a plain page use the path /contact but the contact module registers this path, so pathauto gave my contact page a path of /contact-0. Luckily creating a route subscriber with Drupal Console was painless, so altering the contact module route was very simple to do. I can paste the code here if needed, but most of it is the code that Drupal Console generates for you.

8. PHPunit is bundled into core

Now that Drupal 8 is largely Object Oriented (OO), you are able to test classes using PHPunit. I have wrote about phpunit in the past if you want to know more.

9. Views is in core

This was the main reason why adoption of Drupal 7 was so slow after it’s initial 7.0 release, as everyone needed views to be stable before jumping ship. Now with views bundled into core, views plugins are also being ported at a great rate of knots too.

10. CKEditor is in core

I often found that this was one library that never (or hardly ever) got updated on sites that had been around for a while. More worryingly, CKEditor (the library) would from time to time fix security related issues. Now that this comes with Drupal 8 core, it is just one less thing to worry about.

Also I would love to shout out to Wim Leers (and other contributors) for revamping the image dialog with alignment and caption options. I cannot tell you how much pain and suffering this caused me in Drupal 7.

Comments

If you have built a site recently in Drupal 8 and have found anything interesting or exciting, please let me know in the comments. Also keen to see what sites people have built, so post a link to it if it is public.

Writing PHPunit tests for your custom modules in Drupal 8

Sean Hamlin — Thu, 24 Mar 2016 00:00:00 +1300

Background

I have been doing a bit of Drupal 8 development as of recent, and am loving the new changes, and entities everywhere. I am passionate about automated testing, and when I saw that integrating PHPunit into your custom modules is now even easier, I set out to see how this all worked.

Why is PHPunit important

There are are number of reasons why PHPunit is a great idea

it forces you to write testable code in the first place, this means small classes, with methods that do a single thing
it runs in only a few seconds, there is also no need to have a working Drupal install
integrates with PHPStorm, allowing you run your tests from within the IDE

Step 1, set up your phpunit.xml.dist file

There is a file that comes included with Drupal 8 core, but by default it will not scan any sub-directories under /modules (e.g. like the very common /modules/custom). I stumbled across this question on stackoverflow. So you have a couple of options from here:

Option 1 - Create and use your own phpunit.xml.dist file

You can simply copy (and modify) Drupal 8 core’s phpunit.xml.dist file into git repo somewhere (perhaps outside the webroot), and use this file for all your custom module tests.

Option 2 - Patch Drupal 8 core

Another option (which is the option I took) was to apply a simple patch to Drupal core. There is an open issue on drupal.org to look at scanning all sub-directories for test files. At the time of writing it was uncertain whether this patch would be accepted by the community.

Step 2, write your tests

There are some general conventions you should use when writing your PHPunit tests:

the suffix of the filename should be Test.php, e.g. MonthRangeTest.php
the files should all reside in either the directory /MY_MODULE/tests/src/Unit/ or a sub directory of that

More information on the requirements can be found on the drupal.org documentation.

Dataproviders

Data providers are pretty much the best thing to happen to automated testing. Instead of testing a single scenario, you can instead test a whole range of permutations in order to find those bugs. You start by declaring an annotation @dataProvider for your test method:

<?php
  /**
   * @covers ::getMonthRange
   * @dataProvider monthRangeDataProvider
   */
  public function testGetMonthRange($expected_start, $expected_end, $month_offset, $now) {
    // ... more code
  }

You then declare a method monthRangeDataProvider that returns an array of test cases (which are also arrays). The items in the data provider method are passed one at a time to the testing method, in the same order they are declared (so you can map them to friendly names).

<?php
  /**
   * Data provider for testGetMonthRange().
   *
   * @return array
   *   Nested arrays of values to check:
   *   - $expected_start
   *   - $expected_end
   *   - $month_offset
   *   - $now
   */
  public function monthRangeDataProvider() {
    return [
      // Feb 29 @ noon.
      [1454284800, 1456790399, 0, 1456747200],
      // ... more tests follow
    ];
  }

More information can be found in the phpunit documentation for data providers.

Testing for expected exceptions

Just as important as testing valid inputs, you should also test invalid inputs as well. This is easily achieved with @expectedException annotations above your test method:

<?php
  /**
   * Tests that an end date that is before the start date produces an exception.
   *
   * @expectedException        Exception
   * @expectedExceptionMessage Start date must be before end date
   */
  public function testGetWorkingDaysInRangeException() {
    // ... more code in here
  }

Step 3, enhance your test class with PHPunit metadata

You can annotate both the test class and the methods to provide additional information and metadata about your tests:

@covers

This is mainly used for PHPunit’s automated code coverage report, but I find it also very helpful for developers to up front state what method that are testing.

@coversDefaultClass

This is used at a class level, and saves you having to write rather lengthy @covers statement for all your testing methods, if they all test the same class.

@depends

If a certain test makes no sense to run unless a previous test passed, then you can add in a ‘depends’ annotation above the test method in question. You can depend on multiple other tests too. Note, that this does not change the execution order of the tests, they are still executed top to bottom.

@group or @author

You can think of adding a ‘group’ to a PHPunit class the same as tagging in. It is free tagging in that sense, and you can tag a single class with many tags. This should allow you to categorise your tests. @author is an alias of group, the idea being you can run all tests written by a particular developer.

More information can be found in the PHPunit documentation on annotations.

Step 4, run your test suite

This section assumes you have opted to use Drupal core’s phpunit.xml.dist file (modify the paths as appropriate if you are using a file in another location).

List groups (or tags)

cd core/
../vendor/bin/phpunit --list-groups

Run all tests that are tags with a particular group (or tag)

cd core/
../vendor/bin/phpunit --group tamdash

Example CLI output

$ ../vendor/bin/phpunit --group tamdash
PHPUnit 4.8.11 by Sebastian Bergmann and contributors.
...........
Time: 5.01 seconds, Memory: 144.25Mb
OK (11 tests, 18 assertions)

If you are using PHPStorm, spend a few minutes and set this up too.

Example output

So now there is no need to flip back to your terminal if you just want to quickly run a group of tests.

Conclusion

PHPunit is a great way to be able to run quick tests on isolated parts of your code. Tests often take less than 10 seconds to run, so developer feedback is near instant. It also forces your developers to write better more testible code from the get go. This can only be a good thing. Personally I am very excited to see PHPunit join Drupal 8, and cannot wait to see what people do with it.

Comments

There seems to be quite healthy debate on whether contrib or custom modules should ship with their own phpunit.xml.dist file or whether Drupal core’s file should cover both. I am keen to hear anyone’s thoughts on this. Also let me know if you have any contrib modules in the wild shipping their own phpunit.xml.dist files, and how you found that process.

Deleting multiple fields with large amounts of data in Drupal 7

Sean Hamlin — Wed, 17 Feb 2016 00:00:00 +1300

Background

I was recently tasked with cleaning up a legacy Drupal 7 database to which had accumulated a lot of data in several fields. The fields were no longer used, and the data could be deleted. The data however totalled more than 30 GB with those 7 fields, and this presented a few challenges.

Existing solutions that did not quite cut it

I search around the internet to see the best way to delete fields in Drupal 7 using an update hook. I came across this stack overflow post to which advised using something like:

<?php
function EXAMPLE_update_7001() {
  field_delete_field('field_name');
  field_purge_batch(1);
}
?>

The main issue with the above code is that it only marks the data for deletion, and then slowly removes it on cron. This could take weeks to fully remove the data. It was not an ideal solution in our case.

Solution

If you want to expatiate the process of removing data from fields, and also the fields themselves, we found the code below would remove the 30 GB of data, remove the tables, update the content types, all in one nice update hook. It does rely on having removed the fields from your features, so that are the respective content type can be reverted successfully.

<?php
function EXAMPLE_update_7001() {
  $fields_to_delete = array(
    'field_example_a',
    'field_example_b',
    'field_example_c',
    'field_example_d',
    'field_example_e',
    'field_example_f',
    'field_example_g',
    );
  foreach ($fields_to_delete as $field) {
    db_truncate('field_data_' . $field)->execute();
    db_truncate('field_revision_' . $field)->execute();
    field_delete_field($field);
    watchdog('EXAMPLE', 'Deleted the field @field from all content types.', array('@field' => $field));
  }

  /**
   * The fields aren't really deleted until the purge function runs, ordinarily
   * during cron. Count the number of fields we need to purge, and add five in
   * case a few other miscellaneous fields are in there somehow.
   */
  field_purge_batch(count($fields_to_delete) + 5);

  // Revert features to update the content type in Drupal to drop the field.
  features_revert(array('EXAMPLE' => array('field')));
}

Conclusions

A couple of lessons can be learnt here

Truncations, even on large data sets, are extremely quick, and awesome. The above update hook takes less than 1 minute to run end to end.
If you are going to programmatically update nodes with a high degree of frequency, consider disabling revisions on those content types. Your database thanks you in advance.

Comments

If you have had to delete fields in update hooks in Drupal, let me know if you used another method and how that went for you.

Estimating Drupal projects

Sean Hamlin — Fri, 12 Feb 2016 00:00:00 +1300

Background

Estimating Drupal projects can be tricky, and there are a number of tools and guides out there to help make your life easier in this topic. I thought it would be a great idea to aggregate this data, and present some of the best information out there. It is up to you to choose the best match for your team.

Resources

palantir.net

Developing Drupal sites: Plan or Perish - by Larry Garfield (2013)

Even though this article is nearly 3 years old, it still is completely valid for Drupal. There is an included spreadsheet that you get the tech lead, product owner and another engineer to collaborate on prior to building.

What I like about this:

Emphasis on involving tech leads early to perform scoping and breaking up functionality into tangible Drupal elements. This allows you to have less experienced developers do the actual build, as largely everything has been speced out.
View modes are used extensively which is a great for re-usability of content and maintaining flexibility
Questions, or opportunities for simplification can come up before anything has been built

lullabot.com

There are 3 articles on lullabot.com that are worth reading, they all follow on from one another, so it would pay to start at the beginning and work your way to the end.

The Art of Estimation by Seth Brown (2011)
An Update on the Art of Estimation by Jerad Bitner (2012)
Handling Uncertainty When Estimating Software Projects by Darren Petersen (2015)

In here they introduce a spreadsheet that attempts to combine 2 developers estimates into a single number with variance.

What I like about this:

It exposes the fact that all website builds have range to which it can be completed in
It helps to explain the ‘Wideband Delphi’ method of estimation
Excellent explanation on how to use the spreadsheet is provided

wunderkraut.com

Resources for my session on early estimating by Jakob Persson (2010)

Included in the article here is the Drupal early estimation sheet v3.

What I like about this:

Walks you through an actual example on how an email was turned into a concrete estimate, and how that was analysed
Introduces often overlooked things such as:
- migration of data
- producing help or documentation
- deployments
- third party system integration
- working with a third party (e.g. external design vendor)

drupalize.me

There are several training videos on drupalize.me that would be worth checking out if you have some time:

If you have an existing Acquia subscription you might already have access to Drupalize.me, so this might be worth looking into.

Personal thoughts

Having been involved in doing a lot of this in my previous role, and one additional point I would like to make is to never start from a fresh Drupal install, it often makes sense to standardise on a Drupal technology suite, and installation profiles make this easy.

For instance, Acquia typically use an installation profile called lightning which provides built in ways to do lots of common tasks, e.g. layout, workflow, media management. This saves you having to re-invent the wheel on every project, and should help you provide more solid estimates when it comes to the base features of the site.

Another common theme across the different methods is to involve 1 or 2 tech leads early on to help break the requirements down into Drupal functionality. Having these early conversations with the product owner early on can often lead to a better solution not only in time to build, but feature set too.

Also remember, the above articles and spreadsheets may not work perfectly for you and your organsiation, so feel free to adapt them to suit your needs.

Comments

If you know of some other method to help here, please let me know in the comments. Especially keen to know of any other training content or spreadsheets which other companies use to help estimate Drupal projects.

Measuring Drupal content growth over time

Sean Hamlin — Thu, 03 Dec 2015 00:00:00 +1300

On a recent project I was tasked with working out why the database was so large, in particular certain tables like field_revision_body and field_data_body had grown to be several gigabytes in size.

SQL to the rescue

Here is a simple SQL statement you can execute on your production database to retrieve a count of nodes created grouped by day.

drush --uri=www.example.com sqlq "SELECT count(nid) as count, DATE_FORMAT(FROM_UNIXTIME(created),'%d-%m-%Y') AS month FROM node  GROUP BY YEAR(FROM_UNIXTIME(created)), MONTH(FROM_UNIXTIME(created)), DAY(FROM_UNIXTIME(created))" > /tmp/example-nodes.txt

PostgreSQL users - you can add a custom function to add FROM_UNIXTIME functionality:

CREATE OR REPLACE FUNCTION from_unixtime(integer) RETURNS timestamp AS '
     SELECT to_timestamp($1)::timestamp AS result
' LANGUAGE 'SQL';

Cleaning up results

The SQL will return in a few seconds, and now we need to clean up the data before it is graphed. First lets trim the start of the file, here you can see a few nodes with no valid date, and 1 created in 1980, we want to remove these lines.

head /tmp/example-nodes.txt

count	month
662	NULL
1	08-06-1980
194	30-06-1992
198	01-07-1992
186	02-07-1992
...

Now if we look at the tail you can see a few nodes that are created in the future, I also like to remove these from the results as they tend to blow out the timescale of any graphs:

tail /tmp/example-nodes.txt

1	15-04-2022
12	30-05-2022
1	19-06-2022
1	24-06-2022
1	27-06-2022
...

Conversion to CSV

By default the SQL will return a tab separated values file, and with blank lines in there, lets convert this to CSV and enforce consistent line endings:

cat /tmp/example-nodes.txt | sed $'s/\r//' | tr "\\t" "," > /tmp/example-nodes.txt

Graphing results

Now that you have clean data you can import into your favourite graphing tool, here is Excel making a graph of that data. Here are the number of nodes created by day:

And if you make the total cumulative, you can track total growth in Drupal, and use this to predict into the future:

Wait, what about taxonomy terms

You can extend this method to work with taxonomy terms as well with a slight modification to the SQL (Drupal 7):

drush --uri=www.example.com sqlq "SELECT count(tid) as count, DATE_FORMAT(FROM_UNIXTIME(created),'%d-%m-%Y') AS month FROM taxonomy_term_data GROUP BY YEAR(FROM_UNIXTIME(created)), MONTH(FROM_UNIXTIME(created)), DAY(FROM_UNIXTIME(created))" > /tmp/example-terms.txt

Where to from here

Based on the analysis, you might be able to make recommendations to:

perhaps look at exporting (e.g. with node_export) and removing content from Drupal’s database after X number of years. This will reduce the database size immediately
removing unused revisions from content where the content is older than X number of years
predict into the future how many nodes you will have in 1 year, 5 years etc and be able to plan for this in terms of hardware capacity

Comments

If you are having challenges with Drupal content growing over time, how are you keeping track of this? Are there any other tips and tricks you know about?

Integrating multisite Drupal with Apache Solr

Sean Hamlin — Thu, 15 Oct 2015 00:00:00 +1300

There are a number of ways this can be solved with Drupal 7:

1. Use a search core per site

Using this approach, you create an unique Solr core (N.B. a single Solr application can contain 0 or more cores) per site and either:

Acquia Search Multiple Indexes module (if using Acquia Cloud)
settings.php switch statements to switch on domain

to connect the appropriate search cores to the appropriate Drupal environments. Then you can enable (but not no need to configure) the apachesolr_multisitesearch module.

Pros

Each site’s index is isolate from each other, so there is no interfering with other indexes, and no ability to drop other indexes
You can customise the Solr config per core

Cons

Need to provision a new Solr core every time you add a new site
Need to change settings.php every time you add a new site

2. Use a single shared core, and apachesolr_multisitesearch

In this option you utilise a single shared Solr core, and send the documents from all sites into it. Using special attributes on the documents, you are able to query and get only the results you desire back out again.

The module you need to enable is apachesolr_multisitesearch. Just enabling the module makes your standard search pages return results only for your site. This is accomplished internally by adding a filtering query to each Solr query using hook_apachesolr_query_alter().

You can see the site hashes are different based on the base_url in Drupal already.

Example of this working

Note the site hashes are different:

$ drush --uri=www.example1.com vget apachesolr_site_hash
apachesolr_site_hash: 'lm0ujj'

$ drush --uri=www.example2.com.au vget apachesolr_site_hash
apachesolr_site_hash: '9qt6o6'

Example of this not working

Note the site hashes are the same, thus the search results will be blended:

$ drush --uri=www.example1.com vget apachesolr_site_hash
apachesolr_site_hash: '92ub55'

$ drush --uri=www.example2.com vget apachesolr_site_hash
apachesolr_site_hash: '92ub55'

You will find yourself in this predicament if you clone new sites from old sites (thus the database variable persists).

In order to fix blended site hashes

drop the current index (both sites)
delete the variable ‘apachesolr_site_hash’ (both sites)
visit the admin page solr page (both sites) - this will regenerate the site hash
verify the site hash is now different (both sites)
reindex (both sites)

Pros

No need to provision a new Solr core every time you add a new site
No need to change settings.php every time you add a new site
Hooks into the delete query to prevent dropping the entire index

Cons

You need to ensure the site hash is different for each site, so if you clone one, you will need to delete the variable apachesolr_site_hash straight away
If you ever drop Solr’s index, this will drop all documents across all sites. This should be hard to do (you will need to disable the apachesolr_multisitesearch module)
You must share Solr config (e.g. the schema XML), so you cannot change this per site

Comments

If you run a multisite setup, and make use of Apache Solr, I am keen to hear how you integrated it, and if you have any tips or tricks to share.

Dynamic content caching based on attributes in Drupal 7

Sean Hamlin — Sun, 06 Sep 2015 00:00:00 +1200

Out of the box, Drupal 7 comes with the ability to set the global cache lifetime for all pages on the site. I find this works generally quite well with small sites with not a lot of content or complex caching requirements. Pantheon have a nice write up on the performance settings page that comes with Drupal 7 core and what each of the settings mean.

N.B. In this article I am assuming you are running some form of reverse proxy (e.g. varnish) in front of Drupal (most Drupal managed cloud vendors like Acquia or Pantheon provide varnish as a part of it’s offering as standard).

The problem with Drupal 7 core settings

What happens when you are running a large site, with a very long tail of content? All of a sudden running a global cache TTL of 10 minutes can result in a very poor hit rate in varnish. Having a poor hit rate in varnish ultimately means your web servers end up doing more work, which often leads to having to shell out for additional hardware.

Generally, faced with this issue, you have 2 options:

1) Varnish with purging

In this option, you set a global high cache TTL, so that all content lives in varnish for an excessively long time (say 12 hours), and when certain events occur (say a news article gets published), then purge requests occur telling varnish to drop certain URLs from it’s cache. I don’t want to go into detail to much on this, but I will say there are a number of drawbacks with this approach:

When you publish 1 node, often this node can potentially appear on dozens of pages (e.g. on landing pages as a teaser, or in a view), this means the purge rules get overly complex in a hurry. If you ever re-architect a portion of the site, often this means hundreds of purge rules need to be rewritten and tested, this can be very costly.
Running a CDN (e.g. Akamai, or Cloudflare) can complicate things further, as you need to purge varnish first, then the CDN and never in the reverse order, else the CDN will end up caching the stale content all over again
Having exposed filters on the page means you will need to use bans in varnish (to utlise regex), this is another topic for another day
All too often, content editors resort to nuking the varnish cache from orbit and dropping the whole lot when the rules do not function as desired. This is especially bad as it means all the content in there needs to be entirely rebuilt again. This can (and often does) take down busy sites when it occurs in production.

2) Varnish with intelligent cache TTLs

Another option to which I want to discuss in this blog post is around using the attributes of the node in order to determine at runtime how long this piece of content should be cached in varnish.

Attributes you can use to help you work out the most effective cache TTL:

content type
content created age (now - created)
recency of most recent revision
when the last comment was made
current time of day
any other attribute of the node

What do I mean by this? Well let’s say you run a large digital media organisation, and publish hundreds of fresh news articles per day, but after a week or so, they generally get no edits done to them at all. Your cache TTL rules could be:

if content type == ‘news_article’ then
if age < 1 week, then the cache TTL inherits from the global configuration
else if age > 1 week, then increase cache TTL to 1 week

This way, your new content is able to get upgrades quite promptly, and your long tail of older news articles have a massively inflated cache TTL, meaning your varnish server should really be able to help out here. It is important to note, that these rules apply at a node by node level, and can be made to suit your business requirements.

Modules out there already

I had a look around on Drupal.org to see what others had done in this area in the past, I was genuinely surprised at the lack of modules in this area, here is a short summary:

cacheexclude - used to exclude certain URLs from cache, kind of the opposite of what we want here.
max_age - an (abandoned?) Drupal 6 module that aimed to provide different cache TTLs per content type in addition to the site wide default. This is close to what we want, there even is a patch to help port the module to Drupal 7.

Example code

Rather than using any existing contrib modules, the code required to do what we want is very simple. Here is some simple custom code that can be used as a starting point for your code:

<?php
/**
 * Implements hook_page_build().
 *
 * Responsible for setting the cache TTL based on the content attributes.
 */
function MYMODULE_page_build(&$page) {
  $node = menu_get_object('node');
  if (drupal_page_is_cacheable() && isset($node)) {
    $age_since_last_change = REQUEST_TIME - $node->changed;

    switch ($node->type) {
      case 'news_article' :
        // If the news article has not been edited in a week, then set the cache
        // TTL to a week.
        if ($age_since_last_change > 604800) {
          $max_age = 604800;
          drupal_add_http_header('Cache-Control', 'public,max-age=' . $max_age);
          drupal_add_http_header('Expires', gmdate(DATE_RFC1123, REQUEST_TIME + $max_age));
        }
        break;
    }
  }
}
?>

The above example is simple, but hopefully gives you a taste of what can be achieved with very little code.

How to test your code

I have a few cURL aliases in my .bashrc that will come in handy if you do a lot of this type of thing:

# cURL
function curlh() { curl -sLIXGET "$@"; }
function curlc() { curl -sLIXGET "$@" | grep -E -i "^(Cache-Control|Age|Expires|Set-Cookie|X-Cache|X-Varnish|X-Hits|Vary)"; }

So if you are after all response headers you can:

$ curlh www.example.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html
Date: Sun, 06 Sep 2015 22:52:40 GMT
Etag: "359670651"
Expires: Sun, 13 Sep 2015 22:52:40 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (cpm/F9D5)
X-Cache: HIT
x-ec-custom-error: 1
Content-Length: 1270
Connection: Keep-Alive
Age: 2265

If you just want the relevant caching response headers:

$ curlc www.example.com
Cache-Control: max-age=604800
Expires: Sun, 13 Sep 2015 22:52:40 GMT
X-Cache: HIT
Age: 2249

What this technique relies on

Expectation setting - you will need to ensure all content authors and stakeholders are aware what this means. If a really old news article is edited, there potentially will be lag before the changes are live.
Enough RAM in varnish to house all your objects. If you are RAM constrained, and are seeing a lot of evictions at present, then this technique will not have the impact you will want it to. You might be better to upsize varnish first, then look at this technique.

Comments

I am keen to hear what other large websites do in the area, especially around multiple layers of external cache invalidation and/or if custom cache TTL headers are used. Also if you know of any other contrib modules that can help here, let me know.

Debugging Drupal performance with Cache Debug module

Josh Waihi — Wed, 01 Jul 2015 00:00:00 +1200

Preface

This blog post is for developers, not site builders, as the analysis for cache debugging requires knowledge about the runtime stack of Drupal.

The Problem with Caching in Drupal 7

To obtain performance in Drupal 7, Drupal relies heavily on caching. That is, to process something and cache the end result so that same work doesn’t need to be repeated. Conditions also have to be created for when that cache expires or is invalidated. Drupal has a caching layer to help with this. When you want to store something in cache, you use cache_set, to retrieve it, you use cache_get and to wipe the cache bin clean, you can use cache_clear_all.

Often, modules can implicitly clear or set cache unintentionally. This can lead to more caching overhead than you need. For example, theme registry clearing, use of the variable_set function or calls to other modules that call cache_clear_all. The problem is, how do you track down culprits to fix the issue?

Enter Cache Debug

Cache Debug is a module that wraps around the caching layer and adds logging. Including stacktrace information. It means when a cache_set or cache_clear_all is called, you can trace back to what called it - understand the problem and fix it. Very quickly.

It comes with three logging options:

watchdog - good if you’re using syslog module but deadly if you’re using dblog.
error_log - logs to your php error log
arbitrary file - specify your own log file to log to

Configuring Cache Debug

Because the caching system is so highly utilized, cache logging can be incredibly verbose. Perhaps this is why there is no logging around this in Drupal core. Fortunately, Cache Debug is highly configurable to control what to log.

NOTE: Because the caching system is loaded and used before Drupal’s variable system which manages configuration, it is best to set configuration in settings.php rather than in the database. However, there is a web UI that does set configuration in the database for ease of use.

Basic configuration

If you’ve used the memcache module before, this should feel familiar. In order to use Cache Debug, you need to set it as the cache handler:

<?php
$conf['cache_backends'][] = 'sites/all/modules/cache_debug/cache_debug.inc';
$conf['cache_default_class'] = 'DrupalDebugCache';
?>

This tells Drupal that there is a cache backend located in the path provided (make sure its correct for your Drupal site!) and that the default class for all cache bins is the DrupalDebugCache class. If you only want to monitor a single bin you may want to omit this option.

Since Cache Debug is a logger and not an actual caching system, it needs to pass cache requests onto a real cache system. By default, Debug Cache will use Drupal core’s Database Cache system for cache storage, but if you’re using memcache, redis or similar, you may want to set that as the handler for Cache Debug:

<?php
$conf['cache_debug_class'] = 'MemCacheDrupal';
$conf['cache_cache_debug_form'] = 'DrupalDatabaseCache';
?>

You need to also configure those modules accordingly.

At this point, you’ll be logging all cache calls and stack traces to set and clear calls to the php error log.

Configure the logging location

You may want to choose your own logging location. For example, if you use dblog, then you won’t want to log to watchdog because it will bloat your database. Likewise, if you don’t want to bloat our php error log, then you may want to log to an arbitrary file. You can choose your logging location by setting cache_debug_logging_destination to error_log (default), watchdog or file. For file you will also need to provide the location:

<?php
$conf['cache_debug_logging_destination'] = 'file';
$conf['cache_debug_log_filepath'] = '/tmp/cachedebug.log';
?>

Configuring logging options

You can choose to log calls to cache get, getMulti, set and clear. You can also choose to log a stacktrace of these calls to show the stack that triggered the call. This is most useful for calls to SET and CLEAR. For a minimal logging option with the most about of insight, you might want to try this:

<?php
$conf['cache_debug_log_get'] = FALSE;
$conf['cache_debug_log_getMulti'] = FALSE;
$conf['cache_debug_log_set'] = TRUE;
$conf['cache_debug_log_clear'] = TRUE;
$conf['cache_debug_stacktrace_set'] = TRUE;
$conf['cache_debug_stacktrace_clear'] = TRUE;
?>

Logging per cache bin

You don’t have to log the entire caching layer if you know which bin to look at for the caching issue you’re observing. For example, if you’re looking for misuse of variable_set, you only need to log the cache_bootstrap bin. In which case you could do this:

<?php
# Do not log to all cache bins so ensure this line is removed (from above):
# $conf['cache_default_class'] = 'DrupalDebugCache';
$conf['cache_bootstrap_class'] = 'DrupalDebugCache';
?>

Configure for common issues

Variable set calls and theme registry rebuilds are the two most common issues and so Cache Debug has use cases for these issues built in. So long as Cache Debug is the cache handler for the bin, you can turn off logging and turn on these features and Cache Debug will only log when these issues occur:

<?php
$conf['cache_default_class'] = 'DrupalDebugCache';
$conf['cache_debug_common_settings'] = array(
  'variables' => TRUE,
  'theme_registry' => TRUE,
);
// Turn off logging
$conf['cache_debug_log_get'] = FALSE;
$conf['cache_debug_log_getMulti'] = FALSE;
$conf['cache_debug_log_set'] = FALSE;
$conf['cache_debug_log_clear'] = FALSE;
?>

Analysing the logged data

Cache debug logs to a log file like the example below:

In this snapshot of log output you can see both how cache debug logs cache calls and the stacktracing in action.

Log format structure

A log line starts with a value that describes the cache bin, the cache command and the cache id. E.g. cache_bootstrap->set->variables would bet a cache_set call to the cache_bootstrap cache bin to set the variables cache key. Some calls also log additional data, for example, cache clear also indicates if the call was a wildcard clear. Set calls also log how much data (length) was set.

Stack trace logs

When stack tracing is enabled for specific commands, a stack trace will be logged immediately after the log event that triggered it. The trace rolls back through each function that led to the current cache command being triggered. In the example above you can see that cache_clear_all was called by drupal_theme_rebuild which was called by an include from phptemplate_init. If you look at the source code in phptemplate_init, you’ll see that this means a cache rebuild was triggered from including template.php. In this case it was that Zen base theme had the theme registry rebuild left on.

How to profile PHP memory with Drupal

Sean Hamlin — Wed, 27 May 2015 00:00:00 +1200

Why is this important

How to size the PHP setting max_memory is actually really important for the health of your Drupal application. Size this too small, and you risk getting PHP fatals due to not enough memory allocated. Size this too large, and you are essentially under-utilising your hardware, which in turn can lead to more cost.

How to record every Drupal requests PHP max memory usage

Tim Hillard created this really nice module called Memory profiler, which probably wins some sort of award for being around one of the smallest modules on drupal.org. Essentially this module registers a shutdown function that gets called at the end of every normal Drupal request.

The module is lightweight enough to run on production and only produces an extra syslog line per request.

Analyse the data

The data for memory profiler flows into watchdog, so if you run syslog (which you should), you can use CLI tools to analyse the data.

What does a single request look like

$ grep "memory profiler" drupal-watchdog.log | head -n 1

May 26 06:25:21 10.212.4.16 sitename: https://www.sitename.com|1432621521|memory profiler|1.152.97.17|https://www.sitename.com/|https://www.sitename.com/home|0||4.75 MB - home request_id="v-fc9573dc-036f-11e5-a8c0-22000af91462"

This comes from your syslog format (which can be changed on a per site basis):

$ drush vget syslog_format

syslog_format: '!base_url|!timestamp|!type|!ip|!request_uri|!referer|!uid|!link|!message'

Extract the data from syslog

From here you can tokenise the parts you actually care about, in other words the:

URL requested (part 5)
PHP max memory (part 9)

Using more bash foo

$ grep "memory profiler" drupal-watchdog.log | head -n 1 | awk -F'|' -v OFS=',' '{print $5, $9}'

https://www.sitename.com/,4.75 MB - home request_id="v-fc9573dc-036f-11e5-a8c0-22000af91462"

On Acquia Cloud a request ID is added to all requests, we don’t need this. Also having the string ‘MB’ there is superfluous.

$ grep "memory profiler" drupal-watchdog.log | head -n 1 | awk -F'|' -v OFS=',' '{print $5, $9}' | sed 's/ MB.*//'

https://www.sitename.com/,4.75

Perfect.

So in order to create a CSV for analysing in a spreadsheet you could do:

$ echo "request_uri,max_memory" > /tmp/memory.csv && grep "memory profiler" drupal-watchdog.log | awk -F'|' -v OFS=',' '{print $5, $9}' | sed 's/ MB.*//' >> /tmp/memory.csv

And then you can make pretty graphs if you want:

Or if you just want to find the top requests to your application by memory you can do

$ grep "memory profiler" drupal-watchdog.log | awk -F'|' -v OFS=',' '{print $5, $9}' | sed 's/ MB.*//' | sort -t, -k+2 -n -r | head -n 20

Conclusions

Based on your findings in the logs, you should be able to come up with:

A better understanding of your request memory profile
Better max memory settings for your Drupal application
Potentially identify poor performing pages (memory wise) and can look to optimise them

Gotchas

This module will only work if:

hook_boot() is called (which might not be the case if you run custom lightweight PHP scripts that do not bootstrap Drupal)
The Drupal request is not terminated with a SIGTERM or SIGKILL signal

Comments

Let me know if you found this helpful, or if you have any changes to my bash foo. If you have profiled your Drupal application recently, what methods and tools did you use?

Top 10 DrupalCon LA sessions

Sean Hamlin — Fri, 22 May 2015 00:00:00 +1200

DrupalCon LA

So I did not make it along to DrupalCon Los Angeles, but I did spend some time reading twitter, and watching the sessions online. Here are some of the sessions I found entertaining and insightful and would recommend to others.

Driesnote Keynote

Dries, as always, sets the lay of the land with Drupal. He also goes into the early days of Drupal, and how some key people he was involved with and have now gone on to form organisations that centre around Drupal.

Best quote:

Obstacles don’t block the path, they are the path

No

Larry Garfield gives an interesting talk on why sometimes it is best to say NO in order to give focus to the things that actually matter.

Best quote:

Case and point, the new Macbook Airs, they say NO TO EVERYTHING.

PHP Containers at Scale: 5K Containers per Server

David Strauss explains the history of web hosting, and how this is now far more complex. David is CTO of Pantheon, and they now run 100,000+ websites, all with dev + test + production environments. Pantheon run 150+ containers on a 30GB box (205MB each on average). Really interesting talk on how to run large amounts of sites efficiently.

Decoupled Drupal: When, Why, and How

Amitai Burstein and Josh Koenig give a really entertaining presentation on monolithical architectures and some developer frustrations. And then introduce REST web services in Drupal 8, and how this can be used to provide better consumer interfaces for other frameworks.

Features for Drupal 8

Mike Potter goes through what role features played in Drupal 7, and how features will adapt in Drupal 8 now that CMI is in. Features in Drupal 8 will be going back to it’s roots and provide ‘bundles’ of configuration for re-use.

Meet Commerce 2.x

Ryan and Bojan go through 1.x on Drupal 7, and how they have chosen to develop Commerce 2.x on Drupal 8. This is a complete rewrite. The hierarchical product model is really exciting.

How, When and Why to Patch a Module

Joshua Turton goes over what a patch is, when you should patch contributed modules, and how to keep track of these with Drush make.

My colleague Josh also wrote a blog post on how to use Drush make.

CI for CSS: Creating a Visual Regression Testing Workflow

I topic that I am passionate about is visual regressions, here Kate Kligman goes through some tools that can help you test your site for visual changes. Tools covered include PhantomJS, SlimerJS, Selenium, Wraith.

Speeding up Drupal 8 development using Drupal Console

Eduardo and Jesus give us an introduction to your new best friend in Drupal 8. Drupal console is a Symfony CLI application to help you write boilerplate code, e.g. to create a new module. Personally, I am excited for the form API generator, and the ability to create a new entity with a single command.

For more information see drupalconsole.com.

Q&A with Dries

As Drupal heads down from 130 critical issues down to 22 currently, what are some key concerns by people. The questions are answered by dries, xjm, webchick and alexpott.

Where can I find more videos

Don’t worry there are plenty more videos on the Drupal Association Youtube page.

Comments

If you have any awesome sessions that I have missed let me know in the comments.