Building a GCS Uploader

In my time since joining Google, more than one startup has asked about the possibility of receiving files into Google Cloud Storage (GCS) so that they can be the start of a Pub/Sub based pipeline for all sorts of use cases — image recognition, data ingestion, etc.

To me, this seems like a natural fit for the product. However — there’s no available ‘shim’ layer for doing these — there’s no way to treat GCS like a FTP server, or a direct (simple) web upload target. There are work arounds with gcsfuse or gsutil, but those require end users to install and utilize command line products that are Google specific, and even then have the challenge of not being usable without direct Google credentials in the project.

However, even a naive implementation of such a system would be suboptimal (I suppose thats what its called naive). By placing a single FTP or web upload shim (likely hosted out of a single GCP zone), we negate much of the power and performance of Google’s global network and GCS’s distributed nature, as well as potentially limit performance to the network interface upon which our shim runs. Optimally, we’d want something with the following characteristics:

1. Uses exclusively standard web technologies and runs on modern browsers. This means supporting HTTP/1.1, HTTP/2, TLS, etc.
2. Avoids custom plugins or tools
3. Can be used in an anonymous or semi anonymous way — at minimum we need to avoid the use of project credentials on the client side.
4. Maximizes the value of Google’s network along with the user’s network, so that we get the best upload performance possible.
5. Has the lowest price to maintain — optimally we don’t want to end up with a daemon running somewhere that has a cost much above the cost of GCS object storage in general.

Using these ideas as my guide, I developed a prototype to do exactly that, using only 2 Google products (Cloud Run and Google Cloud Storage).

Ultimately the solution to the problem was to use the fact the GCS supports signed URLs — URLs that have all relevant (time limited) authentication information built into them. We can use those as a way to avoid the need to deliver standard GCP credentials. By using signed URLs we also avoid issues #1, #2 and #4 — all of the bulk upload traffic from the user to GCS is carried directly to the nearest Google Cloud edge point over HTTP, and not through a single system somewhere.

The two biggest challenges are to:

1. Generate those signed URLs
2. Design a frontend UI

Generate Signed URLs

Generating signed URLs within GCP is a fairly simple process, and supported by a wide number of languages and their associated GCP SDKs. Because Python is superior to all other languages in every way, I chose to use it as my base, but it could be done in nearly any language you prefer.

After a few setup procedures detailed in the GitHub repo for this project, I centered on a key function to generate the URLs:

@app.route('/getSignedURL')
def getSignedURL():
    filename = request.args.get('filename')
    action = request.args.get('action')
    blob = bucket.blob(filename)
    url = blob.generate_signed_url(
        expiration=datetime.timedelta(minutes=60),
        method=action, version="v4")
    return url

It’s very simple, and very short — consider this just a backend API. After parsing some incoming parameters (note: there are gaping security holes here — this function uses a client generated value with absolutely no sanity checking), we ask the API to sign a URL for this path in the bucket that’s good for 60 minutes.

It’s worth remembering that the Python itself is just an API — it does not really handle any of client client side code. However, for ease of testing and deployment, I also have the static part of the site (HTML, CSS and JavaScript) served from the same container, but that could be replaced with GCS website serving as a later optimization.

Running this small Python script is its own interesting thought process: Do we run this on a Compute Engine instance, as a AppEngine instance, as a Cloud Function, in Cloud Run or in Kubernetes (GKE)? All of these would work, but for my case, I wanted as little management as possible, making Cloud Functions and Cloud Run the top contenders. Both support scale-to-zero and per-millisecond billing, meeting requirement #5. For me — I’m most comfortable testing, deploying and managing containers, so I went with Cloud Run.

The last important component is the client side work, where the user enters the file they wish to upload with a standard form, then the request for the signed URL is made:

async function generateSignedURL() {
      file = getFilename();
      action = "PUT";
      const response = await fetch('/getSignedURL?filename=' + file + "&action=" + action)
      if (!response.ok) {
        throw new Error('Network response for fetch was not ok.');
      }
      c = await response.text();
      c = c.replace(/\"/g, "")
      console.log("Got signedURL: " + c)
      console.log("Trying to upload " + file)
      upload();
      console.log("Complete")
      return false;
    }

And then lastly the form itself is submitted on button click by the user. This is special because the target of that form is the GCS signed URL directly rather than the Python service, meaning we are only limited by client bandwidth for the upload and maximize the performance benefit of Google’s network.

Once the code has been pushed to Cloud Run (gcloud run deploy), its ready to go (note, I’m skipping the process of building a container — that process is left up to the reader, but the Dockerfile is in the repo)!

You can find the full repo on my GitHub: https://github.com/mcowger/gcs-file-uploader

Building a Simple Web Upload Interface with Google Cloud Run and Cloud Storage was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post discusses the necessary steps to prepare video files and to write the JavaScript logic to play seamless video loops in an Interactive Canvas web app.

Media Source Extensions

Creating seamless video loops requires using Media Source Extensions (MSE), which is a browser feature that extends the HTML media element to allow JavaScript to generate media streams for playback.

You have to write JavaScript code to download and buffer the video data, which is then handed directly to the HTML video tag buffer using MSE. Since MSE is a low-level API, there are a lot of details to get right, but this post will walk through all the necessary steps.

Prepare the video

First, you have to convert the video to the fragmented MP4 file format for streaming. A normal MP4 file consists of a header and the media data. The header is located at the end of the file. For streaming, the header has to be moved to the beginning of the file.

You’ll need the following tools to prepare the video:

• FFmpeg
• MP4Box

Use FFmpeg to convert the file to have the correct code:

ffmpeg -i video.mp4 -an -codec:v libx264 -profile:v baseline 
   -level 3 -b:v 2000k videocodec.mp4

Run the following command to put the header at the front of the file and to ensure that the fragments start with Random Access Points:

MP4Box -dash 1000 -rap -frag-rap videocodec.mp4

A new MP4 file is generated with a “_dashinit” postfix in the filename. Upload this file to your web server.

Play the video

Now that the video file is in the correct format, you’ll use MSE to load and play the file with the HTML media element.

First, define an HTML video element in your web app:

<video id='vid' />

Get a reference to the video element in JavaScript:

let video = document.getElementById('vid');

Create a MediaSource object and create a virtual URL using URL.createObjectURL with the MediaSource object as the source. Then, assign the virtual URL to the media element’s “src” property:

let mediaSource = new MediaSource();

video.src = window.URL.createObjectURL(mediaSource);

Wait for the MediaSource “sourceopen” event to tell you that the media source object is ready for a buffer to be added. Create a SourceBuffer using the MediaSource addSourceBuffer() method with the mime type of the video and then start downloading the file:

let sourceBuffer;
mediaSource.addEventListener('sourceopen', function(){ 
   sourceBuffer = mediaSource.addSourceBuffer(
       'video/mp4; codecs="avc1.42c01e"');
   fileDownload('videocodec_dashinit.mp4');              
});

Use XMLHttpRequest to download the file as an ArrayBuffer:

function fileDownload(url) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', url, true);
  xhr.responseType = 'arraybuffer';
  xhr.send();
  xhr.onload = function(e) {
    if (xhr.status != 200) {
      onLoad();
      return;
    }
    onLoad(xhr.response);
  };
  xhr.onerror = function(e) {
    video.src = null;
  };
};

Append the file data to the SourceBuffer with appendBuffer():

let allSegments;
function onLoad(arrayBuffer) {
  if (!arrayBuffer) {
    video.src = null;
    return;
  }
  allSegments = new Uint8Array(arrayBuffer);
  sourceBuffer.appendBuffer(allSegments);
  processNextSegment();
}

Call the play()method on the video element and append video segments to the source buffer when there is less than 10 seconds left in the playback pipeline:

function processNextSegment() {
  // Wait for the source buffer to be updated
  if (!sourceBuffer.updating &&   
       sourceBuffer.buffered.length > 0) {
    // Only push a new fragment if we are not updating and we have
    // less than 10 seconds in the pipeline
    if (sourceBuffer.buffered.end( 
          sourceBuffer.buffered.length - 1) -  
          video.currentTime < 10) {
      // Append the video segments and adjust the timestamp offset 
      // forward
      sourceBuffer.timestampOffset =  
          sourceBuffer.buffered.end(
              sourceBuffer.buffered.length - 1);
      sourceBuffer.appendBuffer(allSegments);
    }
    // Start playing the video
    if (video.paused) {
      video.play();
    }
  }
  setTimeout(processNextSegment, 1000);
};

The video will keep looping forever and there won’t be any delays between each loop.

Next steps

Even if you don’t understand all of the technical details of using MSE, just copy and paste the code above into an HTML file and try it out in a browser first. Once that is working as expected, add the necessary JavaScript for an Interactive Canvas web app and deploy it as an Action.

It’s also possible to use this technique to create seamless transitions between different videos, but that requires managing a buffer per video.

That’s all you need to display high-production visuals in your Interactive Canvas game. Have fun!

To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.

Use video loops with Interactive Canvas was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you built an Action for the Google Assistant and wondered how many people are using it? Or how many of your users are returning users? In this blog post, we will dive into 3 improvements that the Actions on Google Console team has made to give you more insight into how your Action is being used.

1. Multiple improvements for readability

We’ve updated three areas of the Actions Console for readability: Active Users Chart, Date Range Selection, and Filter Options. With these new updates, you can now better customize the data to analyze the usage of your Actions.

Active Users Chart

The labels at the top of the Active Users chart now read Daily, Weekly and Monthly, instead of the previous 1 Day, 7 Days and 28 Days labels. We also improved the readability of the individual date labels at the bottom of the chart to be more clear. You’ll also notice a quick insight at the bottom of the chart that shows the unique number of users during this time period.

Before

After

Date Range Selection

Previously, the date range selectors applied globally to all the charts. These selectors are now local to each chart, allowing you more control over how you view your data.

The date selector provides the following ranges:

• Daily (last 7 days, last 30 days, last 90 days)
• Weekly (last 4 weeks, last 8 weeks, last 12 weeks, last 24 weeks)
• Monthly (last 3 months, last 6 months, last 12 months)

Filter Options

Previously when you added a filter, it was applied to all the charts on the page. Now, the filters apply to only the chart you’re viewing. We’ve also enhanced the filtering options available for the ‘Surface’ filter, such as mobile devices, smart speakers, and smart display.

Before

After

The filter feature also lets you show data breakdowns over different dimensions. By default, the chart shows a single consolidated line, a result of all the filters applied. You can now select the ‘Show breakdown by’ option to see how the components of that data contribute to the totals based on the dimension you selected.

2. Introducing Retention metrics (New!)

A brand new addition to analytics is the introduction of a retention metrics chart to help you understand how well your action is retaining users. This chart shows you how many users you had in a week and how many returned each week for up to 5 weeks. The higher the percentage week after week, the better your retention.

When you hover over each cell in the chart, you can see the exact number of users who have returned for that week from the previous week.

3. Improvements to Conversation Metrics

Finally, we’ve consolidated the conversation metrics and brought them together into a single chart with separate tabs (‘Conversations’, ‘Messages’, ‘Avg Length’ and ‘Abort rate’) for easier comparison and visibility of trends over time. We’ve also updated the chart labels and tooltips for better interpretation.

Before

After

Next steps

To learn more about what each metric means, you can check out our documentation.

Try out these new improvements to see how your Actions are performing with your users. You can also check out our documentation to learn more. Let us know if you have any feedback or suggestions in terms of metrics that you need to improve your Action. Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

Actions on Google Console Analytics Updates was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Now, let’s look at extending this cloud service to integrate with the Google Assistant through smart home Actions. This enables users to link their account through the Google Home app and control their devices through any Assistant-enabled surface.

If you are unfamiliar with Actions on Google or smart home Actions for the Google Assistant. I recommend reading IoT & Google Assistant part 1 and part 2 by my colleague, Dan Myers, as a starting point.

As a developer, this brings the power of the Home Graph to your devices and gives them context within the user’s home. This context is what enables users to make natural requests, like “What is the temperature in the hallway?”, instead of referring to the device by name.

To build a smart home Action, create a new project in the Actions console. We will add two new features to our device cloud service and configure them in our console project: account linking and intent fulfillment. Let’s start by taking a look at how to integrate with the account linking process.

You can find the sample code described in this post on GitHub.

Account linking

Users authorize the Google Assistant to access their devices through account linking. This process enables the user to sign in to the account they use with the device cloud and connect the device managed by that account to Google. The Actions on Google platform supports several different account linking flows, but only the OAuth 2.0 Authorization Code flow is supported for smart home Actions.

To configure OAuth account linking, you need to supply two endpoints in the Actions console: one for authorization and the other for token exchange. The authorization endpoint is a web UI where the user can authenticate and agree to link their account with the Google Assistant. It must return an authorization code that uniquely identifies the user.

Since we are using Firebase Authentication for the client apps, we can add a new Angular route to our web client for the user to sign-in and return their Firebase ID token as the authorization code once they authorize access:

export class LinkAccountComponent implements OnInit {
  redirectUri: string;
  state: string;
  idToken: string;
  constructor(private authService: AngularFireAuth,
    private route: ActivatedRoute) { }

ngOnInit() {
    this.authService.idToken.subscribe((token) => {
      this.idToken = token;
    });

    this.route.queryParamMap.subscribe((params) => {
      this.redirectUri = params.get('redirect_uri');
      this.state = params.get('state');
    });
  }

linkAccount() {
    const next = new URL(this.redirectUri);
    next.searchParams.append('code', this.idToken);
    next.searchParams.append('state', this.state);
    window.location.href = next.toString();
  }
}

Snippet from web/src/app/link.component.ts

Once the authorization flow is complete, the Google Assistant calls your token endpoint to exchange the authorization code for a persistent refresh token. This token does not expire and remains valid unless the user chooses to revoke device access or unlink their account.

Using the Firebase Admin SDK, we can validate and decode the ID token to obtain the UID of the user. If the token is valid, we can generate a refresh token and associate it with the user’s UID in Firestore. This enables us to look up the token again later for validating future requests.

async function handleAuthorizationCode(request, response) {
  // Auth code is a Firebase ID token
  const decodedToken = await auth.verifyIdToken(request.body.code);
  // Verify UID exists in our database
  const result = await auth.getUser(decodedToken.uid);

  // Encode the user info as a JWT
  const refresh = jwt.sign({
    sub: result.uid,
    aud: client_id
  }, secret);

  // Register this refresh token for the given user
  const userRef = firestore.doc(`users/${result.uid}`);
  await userRef.set({ 'refresh_token': refresh });

  ...
}

Snippet from functions/smart-home/token.js

Firebase Authentication is an identity provider, but not a complete OAuth solution. This means our device cloud service must augment Firebase by minting and verifying tokens used for access to user data. The example code uses the JWT standard to create a self-encoded token with the following JSON payload:

{
  "sub": uid,
  "aud": client_id
  "iat": issued_at_time
}

We are using the JWT.io library for Node.js for all operations related to generating and validating tokens in this example.

The Google Assistant uses this refresh token to request an access token that will authenticate requests for device data. Our example service validates the refresh token signature and checks to make sure it’s the refresh token we expect for that user.

async function handleRefreshToken(request, response) {
  const refreshToken = request.body.refresh_token;
  // Verify UID exists in our database
  const decodedToken = jwt.verify(refreshToken, secret);
  const result = await auth.getUser(decodedToken.sub);

  // Verify incoming token matches our stored refresh token
  const userRef = firestore.doc(`users/${result.uid}`);
  const user = await userRef.get();
  const validToken = user.data().refresh_token;
  if (validToken !== refreshToken) throw new Error(...);

  // Obtain a new access token
  const access = jwt.sign({
    sub: result.uid,
    aud: client_id
  }, secret, {
      expiresIn: '1h'
    });

  ...
}

Snippet from functions/smart-home/token.js

OAuth access tokens should expire, which requires the Assistant service to periodically request a new one using the persistent refresh token. This enables the user to revoke their authorization if necessary. The example access tokens contain the same self-encoded payload as the refresh token, but they expire after one hour.

Intent fulfillment

With the authorization and token endpoints in place, we are ready to begin implementing the fulfillment logic for the user’s devices. In this post, we will focus on implementing each intent in the context of our device cloud example, but you can find additional details on these intents and how they work together in the documentation.

We can use the Actions on Google Client Library for Node.js, which handles parsing the fulfillment requests and provides individual callbacks to handle each intent.

const { smarthome } = require('actions-on-google');
const fulfillment = smarthome();

/** SYNC Intent Handler */
fulfillment.onSync(async (body, headers) => {
  ...
});

/** QUERY Intent Handler */
fulfillment.onQuery(async (body, headers) => {
  ...
});

/** EXECUTE Intent Handler */
fulfillment.onExecute(async (body, headers) => {
  ...
});

/** DISCONNECT Intent Handler */
fulfillment.onDisconnect(async (body, headers) => {
  ...
});

Snippet from functions/smart-home/fulfillment.js

At the beginning of each handler, we need to validate the access token provided with the request. Since the access tokens our application provides are formatted as a JWT that has the user’s UID encoded inside, we simply need to verify the JWT signature using our application’s secret to ensure the token came from us, and check that it has not expired. All of this is handled automatically by the verify() method of the JWT.io client library for Node.js.

const jwt = require('jsonwebtoken');

/**
 * Verify the request credentials provided by the caller.
 * If successful, return UID encoded in the token.
 */
function validateCredentials(headers, jwt_secret) {
  if (!headers.authorization ||
      !headers.authorization.startsWith('Bearer ')) {
    throw new Error('Request missing valid authorization');
  }

  var token = headers.authorization.split('Bearer ')[1];
  var decoded = jwt.verify(token, jwt_secret);

  return decoded.sub;
}

Snippet from functions/smart-home/fulfillment.js

If the provided token is valid, the method will return the UID, which we will need in the intent handlers to query the proper device data. Let’s examine how our device cloud can interact with each intent: SYNC, QUERY, EXECUTE, and DISCONNECT.

SYNC

The Google Assistant sends a SYNC intent after account linking succeeds to request the list of available devices. The response tells the Google Assistant which devices are owned by the given user and their capabilities (also known as traits) of each device. This includes an identifier to represent the user (agentUserId) and a unique id for each device.

For the device cloud sample project, this means returning the list of metadata for all devices where the user’s UID is set as the owner.

fulfillment.onSync(async (body, headers) => {
  const userId = validateCredentials(headers);
  // Return all devices registered to the requested user
  const result = await firestore.collection('devices')
    .where('owner', '==', userId).get();
  const deviceList = [];
  result.forEach(doc => {
    const device = new Device(doc.id, doc.data());
    deviceList.push(device.metadata);
  });

  return {
    requestId: body.requestId,
    payload: {
      agentUserId: userId,
      devices: deviceList
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The SYNC response only contains the device types and their capabilities; it does not report any device state. Below is an example device entry in the SYNC response payload for a light bulb and thermostat:

{
  id: 'light-123abc',
  type: 'action.devices.types.LIGHT',
  traits: [
    'action.devices.traits.OnOff',
    'action.devices.traits.Brightness'
  ],
  name: {
    name: 'Kitchen Light'
  },
  willReportState: true
},
{
  id: 'thermostat-123abc',
  type: 'action.devices.types.THERMOSTAT',
  traits: [
    'action.devices.traits.TemperatureSetting'
  ],
  attributes: {
    availableThermostatModes: 'off,heat,cool',
    thermostatTemperatureUnit: 'C'
  },
  name: {
    name: 'Hallway Thermostat'
  },
  willReportState: true
}

Request Sync

When users add or remove devices associated with their account, you should notify the Google Assistant through the Home Graph API via Request Sync. Without this feature in your service, users must unlink and relink their account to see changes or explicitly say “Hey Google, sync my devices”. Calling the request sync API triggers a new SYNC intent to allow your service to provide updated device information.

In our example, we can observe when a device node is added or removed in Firestore, and request a sync in each instance. The HomeGraph API will throw an error if that user has not linked their account, so we also need to verify that a persisted refresh token exists for the user in Firestore (created during account linking).

const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
  jwt: require('./service-account.json')
});

/**
 * Cloud Function: Request a sync with the Assistant HomeGraph
 * on device add
 */
functions.firestore.document('devices/{device}').onCreate(
  async (snapshot, context) => {
    // Obtain the device owner UID
    const userId = snapshot.data().owner;
    const linked = await verifyAccountLink(userId);
    if (linked) {
      await homegraph.requestSync(userId);
    }
  });

/**
 * Cloud Function: Request a sync with the Assistant HomeGraph
 * on device remove
 */
functions.firestore.document('devices/{device}').onDelete(
  async (snapshot, context) => {
    // Obtain the device owner UID
    const userId = snapshot.data().owner;
    const linked = await verifyAccountLink(userId);
    if (linked) {
      await homegraph.requestSync(userId);
    }
  });

Snippet from functions/smart-home/request-sync.js

QUERY

The QUERY intent asks for the current state of a specific set of devices (noted by their ids). A QUERY may be sent by the Google Assistant in response to a voice command (e.g. “What is the current temperature in the hallway?”) or to update the UI in the Google Home app.

fulfillment.onQuery(async (body, headers) => {
  validateCredentials(headers);
  // Return device state for the requested device ids
  const deviceSet = {};
  for (const target of body.inputs[0].payload.devices) {
    const doc = await firestore.doc(`devices/${target.id}`).get();
    const device = new Device(doc.id, doc.data());
    deviceSet[device.id] = device.reportState;
  }

  return {
    requestId: body.requestId,
    payload: {
      devices: deviceSet
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The device cloud sample project stores this data in the state field for each device using an internal representation of the device attributes. Our QUERY handler converts these attributes to match the device state values required by the Assistant for each trait. Our light bulb and thermostat devices declared support for the following traits:

• OnOff
• Brightness
• TemperatureSetting

Below is an example of the device entries in the QUERY response returning the state for each supported trait:

{
  'light-123abc': {
    online: true,
    on: true,
    brightness: 100
  },
  'thermostat-123abc': {
    online: true,
    thermostatMode: 'heat',
    thermostatTemperatureSetpoint: '20',
    thermostatTemperatureAmbient: '17'
  }
}

EXECUTE

When the user issues a command (e.g. “Turn on the kitchen light”), your service receives an EXECUTE intent. This intent provides a distinct set of traits to be updated for a given set of device ids. This allows a single intent to update a group of traits or devices simultaneously.

Here, we update the contents of the device-configs document for each device, which triggers Cloud IoT Core to publish the configuration change. As we discussed in the previous post, the device will report its new state to Firestore in the devices collection after the change is processed successfully.

fulfillment.onExecute(async (body, headers) => {
  validateCredentials(headers);
  // Update the device configs for each requested id
  const command = body.inputs[0].payload.commands[0];
  
  // Apply the state update to each device
  const update = Device.stateFromExecution(command.execution);
  const batch = firestore.batch();
  for (const target of command.devices) {
    const configRef = firestore.doc(`device-configs/${target.id}`);
    batch.update(configRef, update);
  }
  await batch.commit();

  return {
    requestId: body.requestId,
    payload: {
      commands: {
        ids: command.devices.map(device => device.id),
        status: 'PENDING'
      }
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The EXECUTE response must return a status code indicating whether each command was successful in changing the device state. If the cloud service can synchronously verify that the command reached the device and updated it, then it returns a SUCCESS. If the device is unreachable, then the service can report OFFLINE or ERROR.

Since our commands are written to Firestore through one document and the result is sent asynchronously back through another, we report PENDING rather than reporting SUCCESS. This indicates that we expect the command to succeed, and we will report the state change when it arrives.

Report State

Integrate the Report State API into your service to proactively report changes in device state to the Google Assistant. This is necessary to publish the latest device information to the Home Graph, which enables Google to look up device state without sending additional QUERY intents to your service.

In our example, we can define a new cloud function that triggers on updates to the devices collection. Recall from the previous post that this is where state updates from Cloud IoT Core are published. The function takes the updated device state and forwards it to the Home Graph API.

const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
  jwt: require('./service-account.json')
});

/**
 * Cloud Function: Report device state changes to
 * Assistant HomeGraph
 */
functions.firestore.document('devices/{device}').onUpdate(
  async (change, context) => {
    const deviceId = context.params.device;
    const device = new Device(deviceId, change.after.data());

    // Check if user has linked to Assistant
    const linked = await verifyAccountLink(device.owner);
    if (linked) {
      // Send a state report
      const report = {};
      report[`${device.id}`] = device.reportState;
      await homegraph.reportState({
        requestId: uuid(),
        agentUserId: device.owner,
        payload: {
          devices: {
            states: report
          }
        }
      });
    }
  });

Snippet from functions/smart-home/report-state.js

The format of the states reported for each device is the same as a QUERY response.

DISCONNECT

Your service receives a DISCONNECT intent if the user decides to unlink their account from the Google Assistant. The service should invalidate the credentials used to provide access to this user’s devices.

For our example, this means clearing out the stored refresh token we generated during the account linking process. This negates any future attempts to gain a new access token until the user links their account again.

fulfillment.onDisconnect(async (body, headers) => {
  const userId = validateCredentials(headers);

  // Clear the user's current refresh token
  const userRef = firestore.doc(`users/${userId}`);
  await userRef.delete();

  // Return empty body
  return {};
});

Snippet from functions/smart-home/fulfillment.js

What’s next?

Congratulations! Now your user’s devices are accessible through the Google Assistant. Check out the following resources to go deeper and learn more about building smart home Actions for the Google Assistant:

• Smart Home Device Manager sample
• Smart Home Actions documentation
• Actions on Google client library

You can also follow @ActionsOnGoogle on Twitter and connect with other smart home developers in our Reddit community.

Smart Home Cloud Services with Google: Part 2 was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Testing is an important part of any development process. Without testing, you risk releasing code that results in a frustrating user experience.

Actions on Google are no different. It’s crucial to test all the pieces of your Action to make sure your users will have success and want to come back to your Action.

There are three main components that need to be tested:

• Natural language understanding (NLU), which is how we know what the user wants, or their intent
• Intent handling logic, such as your webhook code that implements response building
• Business logic, such as talking to a database or making external API calls

Testing Actions on Google is a tricky problem because developing an Action spans multiple platforms.

Without test tools, I’ve often had to look through logs just to find out that an intent name I used in Dialogflow mismatched the name I specified in the webhook implementation. In one instance, when trying to figure out why my Action wasn’t working, I realized that the intent I had specified in Dialogflow (“play super fun cats game”) was different than the name I’d put in my webhook (“super cool cats game”). Another time, I just wanted to test a specific intent handler, and my testing process was to play through the conversation and manually observe behavior in the Actions simulator, since there is no direct way to trigger a specific intent.

We’ve heard from developers that the testing experience can be better, so we at Developer Relations have been looking into how to make the testing experience better for developers, and recently we added some testing best practices to our documentation. These summarize our collective findings and are written to help you test your Actions.

The main insight we discovered is that it’s easier to think through your Action if you look at each of the programming layers separately: natural language understanding, intent handling and business logic. This provides a nice mental framework to partition your Action into separate Subjects Under Test (SUT). The diagram below illustrates the layered model.

Testing “Facts about Google”

At Google, we’ve used this mental framework for a few projects already, and in this post I’ll describe my thinking process I used when testing Facts about Google. Despite being an example without significant business logic, “Facts about Google” has non-trivial requirements for NLU: it needs to parse custom entities from user speech.

These custom entities play a crucial role in the Action’s behavior as the Action takes different conversational directions based on those entities. The webhook code has similar requirements and returns complex responses, which are important to how the Action behaves. Those requirements were our base testing requirements.

Dialogflow

First, I’ll take a look at how the testing requirements are implemented for Dialogflow. Because the Action uses Dialogflow to implement NLU, the tests can leverage an API to test the requirements in the Dialogflow fulfillment. Consider the following snippet that uses Dialogflow’s detectIntent method:

test.serial(‘choose_fact’, async function(t) {

const resJson = await dialogflow.detectIntent(

  ‘Tell me about the history of Google’);

expect(resJson.queryResult).to.include.deep.keys(‘parameters’);

// check that Dialogflow extracted required entities from the query.

expect(resJson.queryResult.parameters).to.deep.equal({

  ‘category’: ‘history’,

  // put any other parameters you wish were extracted

});

expect(resJson.queryResult.intent.displayName).to.equal(‘choose_fact’);

 t.pass();

});

This test case asserts that Dialogflow correctly matches a query to an intent and extracts the correct entities (here, entity “category” has value “history”).

Aside: If you’re curious or want additional context, you can refer to df-test.js for the full source code.

I found it useful to map each Dialogflow intent to a test handler (in the snippet above, this is test.serial), which includes assertions applicable for that intent. I recommend testing your NLU for the following:

• Setting entities correctly
• Setting contexts correctly
• Matching difficult queries correctly

“Facts about Google” uses Dialogflow as an implementation of the NLU layer. However, if you’re an Actions SDK developer, you can apply our recommendations to NLU implementation of your choice if it follows similar structured data format.

Webhook

Your webhook plays an important role in controlling a good user experience: it is responsible for conversational responses your Action returns and controls the flow of the conversation.

Facts about Google returns complex responses that include suggestion chips, cards, and text responses. From Google’s conversational design guidelines, it’s known how important it is to incorporate visual responses to better guide the user through conversation. Hence, suggestion chips are an important piece of the Action.

Additionally, one of the common bugs among Actions on Google developers is a result of programmer’s misusing client library by mixing up conv.ask and conv.close. In the snippet below, we test both of those pieces.

The snippet below illustrates how we would test for those bugs. In the code, expect(jsonRes.payload.google.richResponse.suggestions).to.have.deep.members checks that suggestion chips are present and expect(jsonRes.payload.google.expectUserResponse).to.be.true checks that your Action doesn’t close the mic in the middle of conversation with the user.

test.serial(‘yes-history’, async function(t) {

 const jsonRes = await getAppResponse(‘yes-history’);

 expect(jsonRes.payload).to.have.deep.keys(‘google’);

 expect(jsonRes.payload.google.expectUserResponse).to.be.true;

expect(jsonRes.payload.google.richResponse.items).to.have.lengthOf(3);

expect(jsonRes.payload.google.richResponse.suggestions).to.have

   .deep.members([

      {‘title’: ‘Sure’}, {‘title’: ‘No thanks’},

    ]);

 t.pass();

});

The most important piece of the snippet is getAppResponse function that sends a synthetic payload to an instance of your Dialogflow app and receives a response. This response is used as the main SUT. I encourage you to take a look at the detailed implementation in the official repo.

Integration

The tests I wrote for our NLU and business logic layers give us a certain confidence that my Action works as expected, because each unit is tested. However, to boost my confidence even more, I decided to implement an integration test that checks how those two units work together

A good way to come up with a test case is to look at the unit tests done for Dialogflow and webhook, and combine the scenarios. For example, the snippet below combines the test cases we did for Dialogflow and webhook:

test.serial(‘tell me about cats’, async function(t) {

 const jsonRes = await dialogflow.detectIntent(

   ‘Tell me about cats’

);

const payload = jsonRes.queryResult.webhookPayload;

expect(payload).to.have.deep.keys(‘google’);

expect(payload.google.expectUserResponse).to.be.true;

expect(payload.google.richResponse.items)

  .to.have.lengthOf(3);

expect(payload.google.richResponse.suggestions).to.have

  .deep.members([

    {‘title’: ‘Sure’}, {‘title’: ‘No thanks’},

]);

 t.pass();

})

To recap, we used the methodology described in the Testing Best Practices page of the Actions on Google documentation to provide test coverage for “Facts About Google”. One of the goals I had when writing this post was to give insight into my thinking process when coming up with test cases — I encourage you to apply similar processes in your development and provide robust coverage for your Actions.

Thanks for reading! To share your thoughts or questions, join us on Reddit at r/GoogleAssistantDev.

Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on. Can’t wait to see what you build!

Put your Action to the test: Tips to improve your action with testing was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve been using Interactive Canvas for a while, experimenting with various ideas and working with partners to launch their Actions. Along the way, we’ve learned some lessons about what works well, and we’ll pass these on in this post to help you create a successful Action using Interactive Canvas.

Note: At this time, Google is only approving Canvas Actions that are gaming experiences.

Design

Actions using Interactive Canvas are conversational Actions. You should start designing your game by thinking about how a voice user interface can be complemented with visuals using Interactive Canvas. Check out our design guidelines as a starting point for designing your Action conversation.

For Actions using Interactive Canvas, we recommend that you create storyboards to cover all of the main stages of your game, such as the loading screen, the welcome and tutorial screens, the main gaming screens, and the end screen.

Since the user can also touch the screen when playing the game, consider how you’ll provide feedback for these touch events. We recommend that you provide an immediate confirmation within the web app and not wait for the callback response for the user input.

You should also be aware of the various restrictions on Interactive Canvas web apps, like no local storage and staying under the 200MB memory limit.

Once you have determined the visual elements, GUI components, and animation for your Action, select the best technical implementation for the devices that support Interactive Canvas. In the next sections, we recommend various HTML technologies that work well with Interactive Canvas on all the supported devices to create experiences like this demo Action we have been playing with:

Loading page

We recommend using a loading page to wait for the web page to load completely before showing any of your main animations. This will make a noticeable difference to the performance of the animations and avoid any potential layout issues as the page assets are loading.

While the loading page is being displayed, JavaScript logic can be used to asynchronously load any assets like image files and textures.

The simplest way to design a loading page is to have a solid background and display an animated loading indicator. Here, you want to avoid using JavaScript to drive the animation, so using either CSS or an animated GIF is ideal. Take a look at these cool CSS loaders for ideas for your loading page.

Responsive design

Currently, Interactive Canvas is supported on Google Assistant Smart Displays and on Android mobile devices. These devices have different screen resolutions and can also support both landscape and portrait modes.

Your web app should use a responsive design to ensure that the layout and text is appropriately sized for each kind of display. A simple way to do this is to use Sass breakpoints:

@mixin for-small-display-landscape {
    @media screen and (min-width: 1024px) and (min-height: 600px) {
      @content;
    }
}

@mixin for-medium-display-landscape {
    @media screen and (min-width: 1280px) and (min-height: 700px) {
      @content;
    }
}

Also, Actions have a header at the top of the screen that displays the Action name, icon and provides a way for the user to close the action. You shouldn’t place any important content or text behind the header.

Interactive Canvas provides a getHeaderHeightPx() API that asynchronously determines the header height in pixels. Your web app can use this value to dynamically adjust the layout when the web app is loaded:

window.onload = () => {
  const callbacks = {
    onUpdate(data) {
      // update game state based on intent response data
    },
  }
  interactiveCanvas.ready(callbacks)

interactiveCanvas.getHeaderHeightPx().then((height) => {
    // initialize web app layout with header height value
  })
}

Animation

HTML provides a rich set of options to do animation, from manipulating the DOM to CSS animations, HTML canvas, and WebGL.

In our experience, animation that relies on mostly real-time JavaScript calculations tends to require more optimizations. However, there are other options to consider that provide smooth and high FPS.

CSS

Rather than relying on JavaScript to manipulate the DOM, consider using CSS animations. The browser can use the GPU to accelerate these kinds of animations and manage them separately from the main UI thread.

If you don’t want to use the low-level CSS properties, there are CSS libraries like animate.css, which provide CSS classes for a large collection of high-quality animations. To use these animations, first declare your elements with the necessary CSS animation properties:

.title {
  animation-duration: 3s;
  animation-delay: 2s;
  animation-iteration-count: infinite;
}

The animation classes can then be declared statically with the HTML elements or dynamically triggered via JavaScript:

const element = document.querySelector('.title');
element.classList.add('animated', 'bounceInUp');
element.style.visibility = 'visible';
element.addEventListener('animationend', () => {
  element.style.visibility = 'hidden';
});

We recommend that you avoid simultaneous animations and rather use a sequence of animations that are triggered by the end event of the previous animation.

You can also consider using the Web Animations API, which combines the performance of the CSS optimization with the dynamic capabilities of JavaScript.

SVG

SVG allows developers to create two-dimensional vector graphics that can be dynamically manipulated using either JavaScript or CSS.

SVG supports an <animate> tag that can be used to animate an attribute of an element over time. The results on Canvas devices are quite good for basic animations.

However, you can do even better by using CSS to animate SVG graphical elements. The SVG <paths> are assigned classes that are then animated using the CSS animations. These animations are rendered smoothly on devices.

WebGL

WebGL uses the device GPU to significantly improve the performance of animations on web pages. If you do not have experience with WebGL, there are many libraries, like PixiJS, that provide easy, high-level JavaScript APIs for designing animations that are then rendered using WebGL.

For animators who are familiar with tools like Adobe Animate, the PixiAnimate extension allows content to be exported that can then be played by the PixiJS runtime library. In the Canvas web app, the Pixi Animate plugin is used to load the exported animation. JavaScript code can then control the animation and even make dynamic changes to the visuals, like changing the colors of shapes.

We recommend avoiding “Shape Tween” in Adobe Animate since that generates many textures that can run over the memory limit for Canvas web apps. Instead, use “Classic Tweens”.

WebAssembly

WebAssembly (WASM) is an interesting option for developers who have existing code in other languages. For example, you can take your existing game in C or C++ and use a toolchain like Emscripten to compile it to WebAssembly. The Canvas web app can load and instantiate the WASM code. JavaScript logic can then call the WASM functions.

WebAssembly provides developers with low-level control coupled with high-level performance. We’ve experimented with compiling Rust to WASM to avoid garbage collection and achieve super-smooth animations at 60FPS.

Using WASM allows the best of both worlds: you can use HTML for fast text rendering and also displaying static elements, then overlay these on the WASM animations.

Video

Using video assets in a game can be an effective way to display high-production visuals and create engaging experiences such as cutscenes.

For visuals that might be too taxing to generate “on the fly”, consider pre-rendering the animations as a video and then using the HTML media element to play video in the web app. This can be a very effective way to provide rich backgrounds, which are then combined with dynamic DOM elements and animations.

For Canvas, only one active media element is allowed and the video element cannot be styled with CSS.

For long-running effects, the video can be looped by specifying the “loop” attribute for the media element. However, auto looping can introduce a delay between the end of the loop and the beginning of the next loop.

To make the transition seamless requires using the Media Source Extensions (MSE), which extends the media element to allow JavaScript to generate media streams for playback. MSE allows for segments of media to be handed directly to the HTML5 video tag’s buffer. JavaScript code is needed to download and buffer the video data to ensure that the video will keep looping forever and there won’t be any delays between each loop.

Update: Read the post on how use video loops.

State management

The web app can use the Interactive Canvas sendTextQuery() API to invoke an intent in the Dialogflow agent and synchronize its state with the fulfillment logic. This invocation follows the same life-cycle as a user’s vocal response and can add some delays to any backend persistence.

If the timing of persisting any state updates to the backend is time-sensitive, then the web app can invoke any persistence APIs, like Cloud Firestore, directly from its front-end JavaScript logic.

Testing

We recommend using Chrome DevTools to profile your web app performance and memory usage. When you test your Action in the desktop simulator, find the iframe that is used to embed the web app and then use the DevTools to inspect the web app DOM.

We’ve found DevTools useful for profiling the memory and CPU usage. This allows you to find any bottlenecks in your code and improve the FPS of your animations.

We also recommend using stats.js to display an overlay on your game that tracks the real-time FPS.

Once you are happy with the performance, make sure you test your Action on all the devices supported by Interactive Canvas since the screen ratios vary and the device performances differ.

Next steps

Depending on the kind of game you are developing with Interactive Canvas, you have a range of choices to create smooth animations and effects. It does require some testing and profiling, but you can achieve delightful, immersive experiences for your Interactive Canvas games.

It’s still early days for games using Interactive Canvas, so take this opportunity to kick the tires and experiment with what is possible using the power of HTML.

Read our previous post on Interactive Canvas.

To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.

Optimize your web apps for Interactive Canvas was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hackathons are great places to experiment and try new ideas. During my undergraduate years, I attended many hackathons and organized one myself. Everyone gets time to focus on building a prototype, with everything you need to accomplish it. You don’t need to be an expert in a given field, or have access to the highest-end technology, as mentors and MLH are there to help you. It’s a place where you collaborate with others and exchange ideas.

At the end of hacking, you get to show off your project to everyone else. The hackathon is transformed into a tiny world’s fair, where people show off their accomplishments and you get to see everybody’s creativity.

With Google Cloud Platform and Actions on Google, you get access to more resources and tools to bring your ideas to life.

Recap of the hackathon season

To help students get started with Google Cloud Platform, hackathon attendees received Cloud Credits that let them use the assortment of GCP products and APIs to prototype for free. And what differentiates these credits from the GCP free trial is that they don’t require a credit card, a great offering for student populations who may not have one or may be hesitant to provide one for fear of billing snafus.

Additionally, a Best Use of Google Cloud Platform prize is awarded at every hackathon, where each person on the winning team received a Google Home Mini.

The project I Know Trash won the Best Use of Google Cloud Platform prize at DragonHacks, the annual hackathon at Drexel University. It uses the Google Cloud Vision API on a Raspberry Pi to identify the item you’re holding and places it either in the garbage or recycling segments of the trash can. This can allow recycling to be done more efficiently.

Another project, Bus Buddy, won the prize at Cypher, the annual hackathon at William & Mary college. It is an Action for the Google Assistant that provides transit information for the university’s bussing system. You can ask a question like, “When’s the Purple Line coming?” or “When will the Northline arrive at Barracks?” and get quick, immediate answers.

We also provided Google Homes for MLH’s hardware lab, a service that allows attendees to try out some hardware during the event without having to buy it themselves. This makes these services more accessible to a wider audience and allows projects to be innovative without worrying about costs. Hundreds of Google Homes have been checked out during the past year’s hackathon seasons.

Many interesting Actions were created to solve novel challenges. One of them, Doctor Smart Tap, allows one to use voice commands to dispense a precise measurement of liquid. This could be useful in lab environments where you may not want to handle liquids directly.

To learn more about Doctor Smart Tap, you can watch the demo in the video above.

Other Actions, like SickoCode, were more humorous. It uses machine learning to generate rap lyrics in the style of a specific artist.

MLH Localhost

In addition, we worked together to create a Localhost Workshop. MLH Localhost is a program that provides workshops to student groups as content bundles that give them everything they need to host the workshop themselves, teaching the rest of the group on a given topic.

Our workshop, Ok Google, How Do I Build Actions for Assistant, walks students through building an Action that is able to perform sentiment analysis on a given topic by reading recent tweets on that subject. The guide introduces them to Dialogflow and the Cloud Natural Language API to provide a positivity score.

This has been one of the more popular workshops, being run in over 25 workshops from student groups around the world, in places like Parul University and the University of British Columbia.

Conclusion

The Fall 2019 hackathon season begins August 23rd. We can’t wait to see what is built at the next hackathons, and we’ll continue to ensure that you get the resources you need to get started.

Want more? Head over to the Actions on Google Reddit community to discuss Actions with other developers. You can also join the Actions on Google developer community program and you could earn a $200 monthly Google Cloud credit when you publish your first Action.

Training the next generation of voice developers through hackathons and MLH Localhost was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

It was in my second year of college that my friends and I began playing tabletop RPGs. We came up with an elaborate story, and tried to make our games as immersive as possible. The preparation for each session including looking for instrumental music from video games that would fit the mood of each story beat.

In my most recent meetup with my friends to play another board game, I brought my Google Home Mini to provide ambiance and to avoid awkward silences while we thought about our next move.

Building a voice experience for Tabletop Audio made a lot of sense, as it’s easy to start music through a voice command, and Tabletop Audio provides a good collection of audio to listen to based on the situation.

I created an Action for the Google Assistant, and you can now say “Talk to Tabletop Audio” to invoke it and get suggestions. You can then ask questions like “What songs are new”, or start playing a song by saying things like “Play Medieval Fair”. The project has also been set up to support invocation phrases, enabling users to start music with a single query, such as “Ask Tabletop Audio to play Medieval Fair”.

Smart display result for saying “Ask Tabletop Audio to play Medieval Fair”

This is built primarily with media responses, which will let you return an audio file of any length along with track metadata. The Assistant will begin playing this file along with built-in controls on-screen and with your voice. You can pause by tapping on the Google Home Mini, pressing the pause icon, or saying “Hey Google, pause” at any point for the audio to pause. The developer doesn’t have to do any media player logic.

Design

Before writing code, I thought about the different types of interactions that I would want a user to have with this Action, in order to design the conversation appropriately. I started by creating a series of sample dialogs. After I did this, I was able to begin creating intents with Dialogflow.

USER

Hey Google, talk to Tabletop Audio.

GOOGLE ASSISTANT

Sure, here’s Tabletop Audio.

TABLETOP AUDIO

Welcome! I can play a specific track, like ‘Wizard’s Lab, or a genre, like ‘fantasy.’ What would you like?

USER

(Hmm, the party is currently camping.)

Play something with nature.

TABLETOP AUDIO

I found several tracks for the category nature. Here is one at random: Secret Garden.

(Track begins to play)

(A few minutes later)

USER

Hey Google, what song is playing again?

TABLETOP AUDIO

(Music pauses)

This is Secret Garden from Tabletop Audio.

(Music continues)

— -

USER

Hey Google, talk to Tabletop Audio.

GOOGLE ASSISTANT

Sure, here’s Tabletop Audio.

TABLETOP AUDIO

Welcome! I can play a specific track, like ‘Wizard’s Lab’, or a genre, like ‘fantasy.’ What would you like?

USER

What tracks are new?

TABLETOP AUDIO

The last three tracks added to Tabletop Audio are: Wizard’s Lab, Robotics Lab, and Castle Jail. What do you want to listen to?

USER

Castle Jail

TABLETOP AUDIO

Here is Castle Jail

(Music starts playing)

— -

USER

Hey Google, talk to Tabletop Audio.

GOOGLE ASSISTANT

Sure, here’s Tabletop Audio.

TABLETOP AUDIO

Welcome! I can play a specific track, like ‘Wizard’s Lab’, or a genre, like ‘fantasy.’ What would you like?

USER

(Distracted)

No I don’t know where the car keys are

TABLETOP AUDIO

I can’t find a track with that description, but I found others like Cave of Time. You can also ask “What are the latest tracks?”. What would you like to listen to?

USER

Oh, um, play Cave of Time I guess

TABLETOP AUDIO

Here is Cave of Time

(Music starts playing)

Media Responses

First, I needed access to the API for Tabletop Audio. I reached out to the owner with this idea, and he was happy to provide me with a JSON URL that contains an array of every track’s metadata. Once I had access to the API, I was able to begin making queries to get each track’s audio URL and related metadata such as the track name, genre, and image. To prevent a given user from making multiple queries to this API each time they use it, I decided to cache the data when the user first invokes the Action using session data. I can do this using the Node.js Actions on Google library by with the conv.data object. This data will only be available until the Action ends.

As the function was written in TypeScript, I created several interfaces to formally define the data structures I expected. This allows me to take advantage of type-safety through the compiler, and autocompletion in my IDE.

I define the API response in TabletopAudioResponse, which has an array of TabletopAudioTrack objects. I store the API response and the currently playing track in my session data, which is represented by the TabletopAudioSession interface. Then I extend the default DialogflowConversation interface with these types. You can see the implementation in the code snippet below:

To make sure I only call this API once per user per session, I created a function to fetch this data and save it in session data. You can see the implementation of the function below:

When I first prototyped this Action, I only called this function in my welcome intent. After further testing, I found that my Action would not work when I used an invocation phrase because the Action would skip the welcome intent and try to run the playback intent using data that was never fetched.

To make sure that this function would always run the first time that the Action started, I created a middleware function. You can see an implementation in the snippet below:

Now that I had a list of tracks, I was able to create a media response. I abstracted the logic to a separate function, as shown in the snippet below:

My playback intent could then start playing this track:

Now that I had a prototype for playing back audio, it was time to implement a search capability.

Search

Each track has a title, a list of genres, and a list of tags. A user may want to find a track based on any of these parameters, and I wanted to make sure I could capture any of these values and scan the tracklist.

Using Dialogflow, I was able to create a set of training phrases to represent all of the possible ways that a user could search for a given track. I highlighted each searchable term within the phrase and marked it as a parameter with the type @sys.any. This entity type can capture any text, but does mean you should add more training phrases to ensure that Dialogflow can identify what is part of the search.

To make sure that this intent did not trigger too frequently, I changed its priority to Low by selecting the dot in the top-left corner of the page.

By passing the parameter search to my fulfillment, I added a simple search function. If a search query matched a track title, it would begin playing that track. Otherwise, if the user searched for a genre or tag, it would return a random result. I also added some sanitizing of track titles so that users didn’t have to worry about letter case or other symbols. You can see the implementation in the snippet below:

Each song is ten minutes. At the end, the Action receives a callback and it asks a question to the user for the next track to play. Since I know what track is currently playing, I am able to implement a repeat intent which will start playing it from the start. You can see the implementation in the snippet below:

As I began testing this, I saw that some users were not sure whether the Action was still active when they gave a follow-up query after music began to play.

Examining the history of interactions in the Dialogflow console, I saw several instances where someone would say “Ask Tabletop Audio for Outpost 31”. This would then look for a track literally called “Ask Tabletop Audio for Outpost 31” and then tell the user that nothing was found.

With this usage data, I was able to return to the list of training phrases and add several additional phrases to better specify how to extract the search query.

Polished Voice Design

Now that the playback and search were working well, it was time to add extra refinements to make the voice user interface better.

First, I decided to add two additional intents to help guide the user with answers to common questions. One intent would answer about what songs are new, and the other would answer about what this Action could do.

I also took the time to differentiate the visual response from the audible response. On visual surfaces like phones, it’s easy to provide additional information through on-screen suggestion chips or cards. With an audio-only surface like a speaker, I provided a more verbose answer.

To do this, I replaced many of the strings in my conv.ask with a SimpleResponse object that has both a speech and text properties. My Action will speak aloud the speech, while showing the text on the screen. You can see an example of my updated welcome intent below:

Conclusion

The Tabletop Audio Action is a real-world example of how you can build great audio experiences using media responses. With a few small design considerations, you can create a high-quality voice experience that also runs well on phones, smart displays, and many other Assistant surfaces.

To learn more about the implementation of this Action, you can view the source code on GitHub. To learn more about building conversational Actions for the Google Assistant, you can start reading our developer documentation. To learn more about designing high-quality conversational Actions, you can start reading our conversational design best practices documentation.

Want more? Head over to the Actions on Google community on Reddit to discuss Actions with other developers and share what you’ve built on Twitter with the hashtag #AoGDevs. Join the Actions on Google developer community program and you could earn a $200 monthly Google Cloud credit and an Assistant t-shirt when you publish your first app.

Bringing Tabletop Audio to Actions on Google through media responses was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

With Interactive Canvas, you can create Conversational Actions that have rich, full-screen visuals and media using existing web technologies: HTML, JavaScript, CSS, and WebAssembly. This means you don’t need to learn new languages and tools; instead, you can use your favorite web development tools, libraries, and frameworks.

Familiar lifecycle

The typical lifecycle of an Action starts with the user invoking the Action. The Action starts and prompts the user for a response through voice and visuals. This back and forth conversation between the Action and the user continues until the user exits the Action.

The lifecycle of an Interactive Canvas Action is very similar to that of a Conversational Action. The main difference is that an Action can now return an HTML response that loads a web app on devices with displays.

When a user talks to a device with a screen, the Dialogflow NLP matches an intent and then provides an HTML response that loads the web app. The web app initializes the Interactive Canvas API and can update its GUI to match the intent response data.

You now have a new API for the web app to invoke an intent. The flow for that intent is the same as for voice input. This API is useful for triggering an intent in the web app logic to prompt the user for input. Typically, this is used for supporting custom interactive controls on the web app GUI, but could also be called at any time when the Action needs user input.

It’s very easy to add support for the Interactive Canvas API to HTML. All it takes is including our Canvas JavaScript library and declaring a callback for handling the matched intents:

<html>
  <head> 
    <script src="https://www.gstatic.com/assistant/interactivecanvas/api/interactive_canvas.min.js"></script>
  </head>
  <body>
    <script>
      const callbacks = {
        onUpdate(data) {
          // update game state based on intent response data
        },
      };
      interactiveCanvas.ready(callbacks);
    </script>
  </body>
</html>

This way, the state and other data of your game can be synchronized between your backend logic and the web app graphics and animation.

Rich visuals

Interactive Canvas gives you pixel-level control over the rendering of your UI. You’re not forced to use any templates or UI components that we provide, so you can design a visual experience that complements your brand identity by using your own web fonts, background images, color palette, and icons.

You can add visual payoff to conversations by using custom layouts, transitions, and animations. HTML gives you a variety of ways to do animations, including manipulating the DOM or using JavaScript to dynamically control HTML canvas graphics or vector graphics with SVG.

We’ve optimized WebGL to allow for smooth 2D and 3D experiences. In our tests, we have achieved a steady 60 FPS while animating hundreds of sprites.

You can adapt these rich visuals to different devices and screen resolutions by using standard CSS media queries or using your favorite library that supports responsive web design. You can also use the standard HTML orientationchange event to detect the device orientation.

Custom GUIs

You can now create your own custom buttons, tables, and lists. Use one of the many free libraries and frameworks that provide rich interactive components such as Material Web Components, Polymer elements, JQuery UI, or React.

You can also use Standard HTML touch events as an alternative to voice to allow the user to interact with your web app and to select options. For example, you can create your own slideshows and scrolling lists to allow users to explore options provided by your Action.

However, we recommend that you avoid the use of buttons and other GUI elements if there is a better way to use graphics to directly present items or choices. For example, if you want the user to select from one of two game items, show a graphic of each and then highlight the items to make it clear that the user can select either of the items by voice or touch.

Media

Your Interactive Canvas web app can also use powerful media APIs supported by modern web browsers. The HTML media element allows you to play both audio and video. The media element gives you fine-grained control over the playback and allows you to track various media events.

Sound effects can be a great way to provide feedback and set the atmosphere. Delight your users with special sounds when they reach goals in the game or find easter eggs and buried treasure.

Use the Web Audio API to provide background music and feedback. With this API, you can play with effects and filters and mix sounds and music from various sources.

To help you increase the production value of your Action, we provide thousands of sounds in our sound library, which we host for you for free.

You can also use SSML in the intent responses and create layered sounds and music using the <par> tag, which is a unique feature of our platform.

Speech synchronization

Canvas Actions support the standard SSML <mark> tag, which allows you to place custom markers into the text sequence for the TTS:

<speak>
  Go from <mark name="here"/> here,
  to <mark name="there"/> there!
</speak>

As the TTS is played back, events are generated for each mark, which can then be used in your web app logic to synchronize state updates and animations. The platform automatically provides ‘START’ and ‘END’ events for every TTS.

This feature can be used to highlight words as they are spoken or display graphics that match items as they are mentioned in the TTS.

Note: Support for custom marks will be rolling out in the next few months.

Update: We’ve added support for custom marks to synchronize Interactive Canvas animations with SSML events.

Actions features

Canvas Actions can leverage all of the existing APIs and features provided for conversational Actions.

Canvas web apps do have some restrictions like not being able to use cookies or persist data in local storage. However, if you need to persist data within a session, like tracking the number of turns in a game, you can use conv.data within your fulfillment logic. If you want to store data between sessions, like tracking the highest score, then use conv.user.storage to persist data per user. If you need to sync game state across different platforms, you can use a cloud database, for example Firestore.

The Actions simulator fully supports testing your Canvas Actions, and you can use the Chrome DevTools to inspect the DOM, debug your JavaScript code, and optimize the performance.

You can also leverage daily updates, routine suggestions, and notifications to increase engagement with your users.

Earn an income by using our digital purchases API to provide different kinds of digital goods to your users, such as one-time purchases or subscriptions.

Game design

We encourage you to think about new interaction models that weren’t possible on other platforms.

We have a few design ideas to help you be successful with your game:

• Interactive visuals: Take advantage of the visual display in your core game experience by using visual information that users can respond to with their voice.
• Voice forward: Explore experiences where voice is the right input. Good examples include adventure games, visual puzzles solved by voice, and conversations with game characters to unlock new gameplay experiences.
• Shared space device: Smart displays enable local multiplayer games for families to play in a central location. Persistent games allow multiple people to collaborate with a shared game state.

It’s early days for these kinds of games and we encourage you to experiment with new ideas and game designs. But mostly, just have fun! Interactive Canvas is an exciting opportunity for you to create voice-enabled games that are complemented with rich visuals and implemented in a language you already know — regular HTML.

Next steps

To get started with Canvas, take a look at our basic sample or our simple game that we have open sourced on Github. Make sure to read our docs and watch our introductory video to learn about the basics of Interactive Canvas.

Let the games begin!

Read our next post on Interactive Canvas.

To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.

Create rich, immersive Google Assistant Games with Interactive Canvas was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.