Now, let’s look at extending this cloud service to integrate with the Google Assistant through smart home Actions. This enables users to link their account through the Google Home app and control their devices through any Assistant-enabled surface.

If you are unfamiliar with Actions on Google or smart home Actions for the Google Assistant. I recommend reading IoT & Google Assistant part 1 and part 2 by my colleague, Dan Myers, as a starting point.

As a developer, this brings the power of the Home Graph to your devices and gives them context within the user’s home. This context is what enables users to make natural requests, like “What is the temperature in the hallway?”, instead of referring to the device by name.

To build a smart home Action, create a new project in the Actions console. We will add two new features to our device cloud service and configure them in our console project: account linking and intent fulfillment. Let’s start by taking a look at how to integrate with the account linking process.

You can find the sample code described in this post on GitHub.

Account linking

Users authorize the Google Assistant to access their devices through account linking. This process enables the user to sign in to the account they use with the device cloud and connect the device managed by that account to Google. The Actions on Google platform supports several different account linking flows, but only the OAuth 2.0 Authorization Code flow is supported for smart home Actions.

To configure OAuth account linking, you need to supply two endpoints in the Actions console: one for authorization and the other for token exchange. The authorization endpoint is a web UI where the user can authenticate and agree to link their account with the Google Assistant. It must return an authorization code that uniquely identifies the user.

Since we are using Firebase Authentication for the client apps, we can add a new Angular route to our web client for the user to sign-in and return their Firebase ID token as the authorization code once they authorize access:

export class LinkAccountComponent implements OnInit {
  redirectUri: string;
  state: string;
  idToken: string;
  constructor(private authService: AngularFireAuth,
    private route: ActivatedRoute) { }

ngOnInit() {
    this.authService.idToken.subscribe((token) => {
      this.idToken = token;
    });

    this.route.queryParamMap.subscribe((params) => {
      this.redirectUri = params.get('redirect_uri');
      this.state = params.get('state');
    });
  }

linkAccount() {
    const next = new URL(this.redirectUri);
    next.searchParams.append('code', this.idToken);
    next.searchParams.append('state', this.state);
    window.location.href = next.toString();
  }
}

Snippet from web/src/app/link.component.ts

Once the authorization flow is complete, the Google Assistant calls your token endpoint to exchange the authorization code for a persistent refresh token. This token does not expire and remains valid unless the user chooses to revoke device access or unlink their account.

Using the Firebase Admin SDK, we can validate and decode the ID token to obtain the UID of the user. If the token is valid, we can generate a refresh token and associate it with the user’s UID in Firestore. This enables us to look up the token again later for validating future requests.

async function handleAuthorizationCode(request, response) {
  // Auth code is a Firebase ID token
  const decodedToken = await auth.verifyIdToken(request.body.code);
  // Verify UID exists in our database
  const result = await auth.getUser(decodedToken.uid);

  // Encode the user info as a JWT
  const refresh = jwt.sign({
    sub: result.uid,
    aud: client_id
  }, secret);

  // Register this refresh token for the given user
  const userRef = firestore.doc(`users/${result.uid}`);
  await userRef.set({ 'refresh_token': refresh });

  ...
}

Snippet from functions/smart-home/token.js

Firebase Authentication is an identity provider, but not a complete OAuth solution. This means our device cloud service must augment Firebase by minting and verifying tokens used for access to user data. The example code uses the JWT standard to create a self-encoded token with the following JSON payload:

{
  "sub": uid,
  "aud": client_id
  "iat": issued_at_time
}

We are using the JWT.io library for Node.js for all operations related to generating and validating tokens in this example.

The Google Assistant uses this refresh token to request an access token that will authenticate requests for device data. Our example service validates the refresh token signature and checks to make sure it’s the refresh token we expect for that user.

async function handleRefreshToken(request, response) {
  const refreshToken = request.body.refresh_token;
  // Verify UID exists in our database
  const decodedToken = jwt.verify(refreshToken, secret);
  const result = await auth.getUser(decodedToken.sub);

  // Verify incoming token matches our stored refresh token
  const userRef = firestore.doc(`users/${result.uid}`);
  const user = await userRef.get();
  const validToken = user.data().refresh_token;
  if (validToken !== refreshToken) throw new Error(...);

  // Obtain a new access token
  const access = jwt.sign({
    sub: result.uid,
    aud: client_id
  }, secret, {
      expiresIn: '1h'
    });

  ...
}

Snippet from functions/smart-home/token.js

OAuth access tokens should expire, which requires the Assistant service to periodically request a new one using the persistent refresh token. This enables the user to revoke their authorization if necessary. The example access tokens contain the same self-encoded payload as the refresh token, but they expire after one hour.

Intent fulfillment

With the authorization and token endpoints in place, we are ready to begin implementing the fulfillment logic for the user’s devices. In this post, we will focus on implementing each intent in the context of our device cloud example, but you can find additional details on these intents and how they work together in the documentation.

We can use the Actions on Google Client Library for Node.js, which handles parsing the fulfillment requests and provides individual callbacks to handle each intent.

const { smarthome } = require('actions-on-google');
const fulfillment = smarthome();

/** SYNC Intent Handler */
fulfillment.onSync(async (body, headers) => {
  ...
});

/** QUERY Intent Handler */
fulfillment.onQuery(async (body, headers) => {
  ...
});

/** EXECUTE Intent Handler */
fulfillment.onExecute(async (body, headers) => {
  ...
});

/** DISCONNECT Intent Handler */
fulfillment.onDisconnect(async (body, headers) => {
  ...
});

Snippet from functions/smart-home/fulfillment.js

At the beginning of each handler, we need to validate the access token provided with the request. Since the access tokens our application provides are formatted as a JWT that has the user’s UID encoded inside, we simply need to verify the JWT signature using our application’s secret to ensure the token came from us, and check that it has not expired. All of this is handled automatically by the verify() method of the JWT.io client library for Node.js.

const jwt = require('jsonwebtoken');

/**
 * Verify the request credentials provided by the caller.
 * If successful, return UID encoded in the token.
 */
function validateCredentials(headers, jwt_secret) {
  if (!headers.authorization ||
      !headers.authorization.startsWith('Bearer ')) {
    throw new Error('Request missing valid authorization');
  }

  var token = headers.authorization.split('Bearer ')[1];
  var decoded = jwt.verify(token, jwt_secret);

  return decoded.sub;
}

Snippet from functions/smart-home/fulfillment.js

If the provided token is valid, the method will return the UID, which we will need in the intent handlers to query the proper device data. Let’s examine how our device cloud can interact with each intent: SYNC, QUERY, EXECUTE, and DISCONNECT.

SYNC

The Google Assistant sends a SYNC intent after account linking succeeds to request the list of available devices. The response tells the Google Assistant which devices are owned by the given user and their capabilities (also known as traits) of each device. This includes an identifier to represent the user (agentUserId) and a unique id for each device.

For the device cloud sample project, this means returning the list of metadata for all devices where the user’s UID is set as the owner.

fulfillment.onSync(async (body, headers) => {
  const userId = validateCredentials(headers);
  // Return all devices registered to the requested user
  const result = await firestore.collection('devices')
    .where('owner', '==', userId).get();
  const deviceList = [];
  result.forEach(doc => {
    const device = new Device(doc.id, doc.data());
    deviceList.push(device.metadata);
  });

  return {
    requestId: body.requestId,
    payload: {
      agentUserId: userId,
      devices: deviceList
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The SYNC response only contains the device types and their capabilities; it does not report any device state. Below is an example device entry in the SYNC response payload for a light bulb and thermostat:

{
  id: 'light-123abc',
  type: 'action.devices.types.LIGHT',
  traits: [
    'action.devices.traits.OnOff',
    'action.devices.traits.Brightness'
  ],
  name: {
    name: 'Kitchen Light'
  },
  willReportState: true
},
{
  id: 'thermostat-123abc',
  type: 'action.devices.types.THERMOSTAT',
  traits: [
    'action.devices.traits.TemperatureSetting'
  ],
  attributes: {
    availableThermostatModes: 'off,heat,cool',
    thermostatTemperatureUnit: 'C'
  },
  name: {
    name: 'Hallway Thermostat'
  },
  willReportState: true
}

Request Sync

When users add or remove devices associated with their account, you should notify the Google Assistant through the Home Graph API via Request Sync. Without this feature in your service, users must unlink and relink their account to see changes or explicitly say “Hey Google, sync my devices”. Calling the request sync API triggers a new SYNC intent to allow your service to provide updated device information.

In our example, we can observe when a device node is added or removed in Firestore, and request a sync in each instance. The HomeGraph API will throw an error if that user has not linked their account, so we also need to verify that a persisted refresh token exists for the user in Firestore (created during account linking).

const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
  jwt: require('./service-account.json')
});

/**
 * Cloud Function: Request a sync with the Assistant HomeGraph
 * on device add
 */
functions.firestore.document('devices/{device}').onCreate(
  async (snapshot, context) => {
    // Obtain the device owner UID
    const userId = snapshot.data().owner;
    const linked = await verifyAccountLink(userId);
    if (linked) {
      await homegraph.requestSync(userId);
    }
  });

/**
 * Cloud Function: Request a sync with the Assistant HomeGraph
 * on device remove
 */
functions.firestore.document('devices/{device}').onDelete(
  async (snapshot, context) => {
    // Obtain the device owner UID
    const userId = snapshot.data().owner;
    const linked = await verifyAccountLink(userId);
    if (linked) {
      await homegraph.requestSync(userId);
    }
  });

Snippet from functions/smart-home/request-sync.js

QUERY

The QUERY intent asks for the current state of a specific set of devices (noted by their ids). A QUERY may be sent by the Google Assistant in response to a voice command (e.g. “What is the current temperature in the hallway?”) or to update the UI in the Google Home app.

fulfillment.onQuery(async (body, headers) => {
  validateCredentials(headers);
  // Return device state for the requested device ids
  const deviceSet = {};
  for (const target of body.inputs[0].payload.devices) {
    const doc = await firestore.doc(`devices/${target.id}`).get();
    const device = new Device(doc.id, doc.data());
    deviceSet[device.id] = device.reportState;
  }

  return {
    requestId: body.requestId,
    payload: {
      devices: deviceSet
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The device cloud sample project stores this data in the state field for each device using an internal representation of the device attributes. Our QUERY handler converts these attributes to match the device state values required by the Assistant for each trait. Our light bulb and thermostat devices declared support for the following traits:

• OnOff
• Brightness
• TemperatureSetting

Below is an example of the device entries in the QUERY response returning the state for each supported trait:

{
  'light-123abc': {
    online: true,
    on: true,
    brightness: 100
  },
  'thermostat-123abc': {
    online: true,
    thermostatMode: 'heat',
    thermostatTemperatureSetpoint: '20',
    thermostatTemperatureAmbient: '17'
  }
}

EXECUTE

When the user issues a command (e.g. “Turn on the kitchen light”), your service receives an EXECUTE intent. This intent provides a distinct set of traits to be updated for a given set of device ids. This allows a single intent to update a group of traits or devices simultaneously.

Here, we update the contents of the device-configs document for each device, which triggers Cloud IoT Core to publish the configuration change. As we discussed in the previous post, the device will report its new state to Firestore in the devices collection after the change is processed successfully.

fulfillment.onExecute(async (body, headers) => {
  validateCredentials(headers);
  // Update the device configs for each requested id
  const command = body.inputs[0].payload.commands[0];
  
  // Apply the state update to each device
  const update = Device.stateFromExecution(command.execution);
  const batch = firestore.batch();
  for (const target of command.devices) {
    const configRef = firestore.doc(`device-configs/${target.id}`);
    batch.update(configRef, update);
  }
  await batch.commit();

  return {
    requestId: body.requestId,
    payload: {
      commands: {
        ids: command.devices.map(device => device.id),
        status: 'PENDING'
      }
    }
  };
});

Snippet from functions/smart-home/fulfillment.js

The EXECUTE response must return a status code indicating whether each command was successful in changing the device state. If the cloud service can synchronously verify that the command reached the device and updated it, then it returns a SUCCESS. If the device is unreachable, then the service can report OFFLINE or ERROR.

Since our commands are written to Firestore through one document and the result is sent asynchronously back through another, we report PENDING rather than reporting SUCCESS. This indicates that we expect the command to succeed, and we will report the state change when it arrives.

Report State

Integrate the Report State API into your service to proactively report changes in device state to the Google Assistant. This is necessary to publish the latest device information to the Home Graph, which enables Google to look up device state without sending additional QUERY intents to your service.

In our example, we can define a new cloud function that triggers on updates to the devices collection. Recall from the previous post that this is where state updates from Cloud IoT Core are published. The function takes the updated device state and forwards it to the Home Graph API.

const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
  jwt: require('./service-account.json')
});

/**
 * Cloud Function: Report device state changes to
 * Assistant HomeGraph
 */
functions.firestore.document('devices/{device}').onUpdate(
  async (change, context) => {
    const deviceId = context.params.device;
    const device = new Device(deviceId, change.after.data());

    // Check if user has linked to Assistant
    const linked = await verifyAccountLink(device.owner);
    if (linked) {
      // Send a state report
      const report = {};
      report[`${device.id}`] = device.reportState;
      await homegraph.reportState({
        requestId: uuid(),
        agentUserId: device.owner,
        payload: {
          devices: {
            states: report
          }
        }
      });
    }
  });

Snippet from functions/smart-home/report-state.js

The format of the states reported for each device is the same as a QUERY response.

DISCONNECT

Your service receives a DISCONNECT intent if the user decides to unlink their account from the Google Assistant. The service should invalidate the credentials used to provide access to this user’s devices.

For our example, this means clearing out the stored refresh token we generated during the account linking process. This negates any future attempts to gain a new access token until the user links their account again.

fulfillment.onDisconnect(async (body, headers) => {
  const userId = validateCredentials(headers);

  // Clear the user's current refresh token
  const userRef = firestore.doc(`users/${userId}`);
  await userRef.delete();

  // Return empty body
  return {};
});

Snippet from functions/smart-home/fulfillment.js

What’s next?

Congratulations! Now your user’s devices are accessible through the Google Assistant. Check out the following resources to go deeper and learn more about building smart home Actions for the Google Assistant:

• Smart Home Device Manager sample
• Smart Home Actions documentation
• Actions on Google client library

You can also follow @ActionsOnGoogle on Twitter and connect with other smart home developers in our Reddit community.

Smart Home Cloud Services with Google: Part 2 was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the article, Dan mentions that in order to integrate your devices with Google Assistant, “you, as the device creator, develop your own cloud service, which includes its own dashboard, device registration, and device management that functions independently of the Assistant.”

In this post, I’d like to explore what that cloud service might look like if you’re a developer who hasn’t already invested the time and resources into building your own device cloud — or simply don’t want to manage the cloud infrastructure yourself as your device fleet scales. Maybe you’re just looking into getting your existing products connected, and wondering what it takes to build a cloud service for the smart home.

We’re going to discuss a working example of a smart home cloud using components of Firebase and the Google Cloud Platform. You will see how to securely connect and provision devices, build a user-authenticated device model that enables easy front-end user application development for both mobile and the web, and set up the structure necessary to integrate with the Google Assistant.

You can find the sample code described in this post on GitHub.

Inside the blue box

Expanding a bit on what a smart home device cloud should look like, here are some typical requirements you might have:

• Send commands to devices from the cloud
• Report state from devices to the cloud
• Detect when devices are offline
• Provision devices to individual users
• Provide users access to manage their devices

Not to mention that you want all of this to happen securely as data travels between users, the cloud, and devices. We can satisfy all of these requirements using services from the Google Cloud Platform and Firebase to create a scalable serverless solution for home devices.

Cloud IoT Core is a fully managed service for securely connecting and managing IoT devices. Using the MQTT or HTTP bridge, IoT devices can connect to Google Cloud using per-device public/private key authentication and exchange data. Incoming device data is published to a Cloud Pub/Sub event stream.

Cloud Firestore is a flexible, scalable NoSQL cloud database to store and sync data for client- and server-side development. It keeps your data in sync across client apps through realtime listeners and offers offline support for mobile and web through their native SDKs. Firestore also pairs with Firebase Authentication to control access to data through built-in security rules.

Cloud Functions enable backend code to run in response to events triggered by Firebase and Google Cloud features. We can use this to marshal device data between IoT Core and Firestore. Using this architecture, Cloud Firestore becomes the central data hub and source of truth for the state of all connected devices and exposes that state to authenticated client applications.

Device Model

The Firestore model represents data as key-value pairs stored in documents, which are then organized into collections. We will represent each smart home device in Firestore using two documents:

1. Document within a devices collection containing the device metadata, online status, and latest state reported by the device.
2. Document within a device-configs collection with the user’s latest requested state (e.g. “set the thermostat to 75 degrees”).

devices/light-123abc
  name: "Kitchen Light"
  owner: $user-id
  type: "light"
  online: true
  state: {
    on: true
    brightness: 100
  }

device-configs/light-123abc
  owner: $user-id
  value: {
    on: true
    brightness: 100
  }

devices/thermostat-123abc
  name: "Hallway Thermostat"
  owner: $user-id
  type: "thermostat"
  online: true
  state: {
    mode: heat
    current: 70
    setpoint: 72
  }

device-configs/thermostat-123abc
  owner: $user-id
  value: {
    on: true
    setpoint: 72
  }

Changes to the device’s config document trigger a cloud function to update the device. By separating the configuration and state into two documents, we ensure that successful updates received from the device don’t trigger the same logic. This simplifies the function logic and creates a nice separation outgoing and incoming data flows.

Device Communication

We need to map changes in state to messages that we can exchange with the device. Cloud IoT Core supports the following message types:

• Configuration: Sent from the cloud to the device, up to one message per second. Configuration messages are guaranteed to be delivered to the device.
• Commands: Send up to 100 messages per second from the cloud to the device. Commands are only delivered if the device is online.
• State: Sent from the device to the cloud, up to one message per second. State updates are delivered to active Pub/Sub subscribers and recent updates are persisted inside Cloud IoT Core.
• Telemetry: Send up to 100 messages per second from the device to the cloud. Telemetry events are only delivered to active Pub/Sub subscribers.

We want user commands to turn on the lights or adjust the temperature to be guaranteed to arrive at the device, even if that device happens to be offline. With configuration messages, this is handled for us by IoT Core. The device will receive the latest configuration each time it connects to the gateway, removing the need for our application logic to handle this.

Using cloud functions, we can deploy code that triggers when a device config document changes in Firestore, and publish those to the corresponding device in IoT Core. The document path used as a trigger contains a wildcard for the device id. This allows the function to trigger for every device, and captures the device id value in the function’s context.

functions.firestore.document('device-configs/{device}').onWrite(
  async (change, context) => {
    const deviceId = context.params.device;
    const config = change.after.data();

    ...

    // Create a new Cloud IoT client
    const client = google.cloudiot({
      version: 'v1',
      auth: auth
    });

    // Update IoT Core configuration
    const parent =
      'projects/my-project/locations/us-central1';
    const devicePath = `${parent}/registries/my-registry`;
    const request = {
      name: `${devicePath}/devices/${deviceId}`,
      binaryData: Buffer.from(JSON.stringify(config))
        .toString("base64"),
    };
    await client.projects.locations.registries.devices
      .modifyCloudToDeviceConfig(request);
  });

When the device reports its state back to IoT Core, the message gets published to a Cloud Pub/Sub topic that we choose. We can then use another function to capture those messages and write the updated state into Firestore.

functions.pubsub.topic('device-events').onPublish(
  async (message) => {
    const deviceId = message.attributes.deviceId;

    // Write the device state into Firestore
    const deviceRef = firestore.doc(`devices/${deviceId}`);
    try {
      await deviceRef.update({
        'state': message.json,
        'online': true
      });
      console.log(`State updated for ${deviceId}`);
    } catch (error) {
      console.error(error);
    }
});

Our devices could publish this data as either state updates or telemetry events, and the code would behave the same way. However, because Firestore does the work of persisting device state in our cloud service, we don’t need IoT Core to do the same. Therefore, it makes the most sense to have the device send data using telemetry events.

NOTE: The sample-device project on GitHub implements a virtual device using Node.js that connects to the MQTT gateway, subscribes to configuration changes, and publishes updates back as telemetry events.

Building the Clients

Because all of the device data resides in Firestore, building user applications for the web and mobile devices is very straightforward using the native Firebase SDKs. To make things even simpler, the Firebase team provides additional libraries that integrate these SDKs with popular frameworks.

This example uses Angular and AngularFire for the web interface. The mobile application was built in Flutter with FlutterFire, which has the added benefit of being cross-platform between iOS and Android devices.

Security rules define access control for the data housed in Firestore, ensuring only authorized users have permission to view and manage the devices they own. The rule set below defines the following controls:

1. Users can read, change, or remove a device where they are listed as the owner
2. Users cannot create a new device entry (this will be handled by device provisioning)
3. Users can create and modify a pending device entry (part of device provisioning)

service cloud.firestore {
  match /databases/{database}/documents {
    match /devices/{deviceid} {
      allow read, update, delete:
       if request.auth.uid == resource.data.owner;
    }
    
    match /device-configs/{deviceid} {
      allow read, update, delete:
       if request.auth.uid == resource.data.owner;
    }
    
    match /pending/{deviceid} {
      allow read, write:
        if request.auth.uid == resource.data.owner;
      allow create:
        if request.auth.uid != null;
    }
  }
}

Since Firestore represents the source of truth for device data, client applications only need to interface with the Firebase SDK to authenticate users and manage devices. Users can view their devices by listing the documents in the devices collection where their account matches the owner field. The following Dart code lists the documents for the current user’s devices using the FlutterFire plugin.

Firestore.instance.collection('devices')
  .where('owner', isEqualTo: user.uid)
  .snapshots()

Each user command to change the state of the device updates the corresponding document in the device-configs collection. This triggers an update to the device’s IoT Core configuration using the cloud function described in the previous section.

Firestore.instance.collection('device-configs')
  .document(device.id)
  .updateData({
    'value': ...
  });

Registering a new device to the user’s account requires a few additional steps for security purposes, so let’s look in more detail at that process.

Device Provisioning

Cloud IoT Core uses public/private key pairs to authenticate devices. We won’t discuss the process in detail here, but see device security for more information on how it works. For consumer devices, we will split the provisioning process into two stages: factory provisioning and end-user provisioning.

The factory assigns a private key to the physical device, and registers the corresponding public key and device identifier with Cloud IoT Core. By doing this at the factory, we ensure only devices we manufacture have credentials to access the cloud. The device is shipped to the end user with a copy of the public key, which the user application must provide in order to register that device to their account.

In our example, the public key and device metadata are bound to a QR code that the user can scan from their mobile device during the device registration process. The example QR code contains the following information as a JSON blob:

{
  "type":"light",
  "serial_number":"abcdef123456",
  "public_key":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE4InTLsvDq9Km..."
}

The user application writes this data into Firestore as a pending device registration, which triggers a cloud function to verify that the public key from the physical device matches the value provisioned in Cloud IoT Core for the corresponding device identifier and that the device is not already registered to a different account.

functions.firestore.document('pending/{device}').onWrite(
  async (change, context) => {
    const deviceId = context.params.device;
    const pending = change.after.data();
    
    ...

    // Create a new Cloud IoT client
    const client = google.cloudiot({
      version: 'v1',
      auth: auth
    });
    const parent = 'projects/my-project/locations/us-central1';
    const devicePath = `${parent}/registries/my-registry`;
    const request = {
      name: `${devicePath}/devices/${deviceId}`
    }

    try {
      // Verify device does NOT already exist in Firestore
      const deviceRef = firestore.doc(`devices/${deviceId}`);
      const deviceDoc = await deviceRef.get();
      if (deviceDoc.exists)
        throw `${deviceId} is already registered to another user`;

      // Verify device exists in IoT Core
      // Throws an error if device id does not exist
      const result = await client.projects.locations.registries
        .devices.get(request);

      // Verify the device public key matches
      const deviceKey = result.credentials[0].publicKey.key
        .trim();
      const pendingKey = pending.public_key;
      if (deviceKey !== pendingKey) throw 'Public Key Mismatch';

      ...

    } catch (error) {
      console.error(error);
    }
  });

Once the device data has been verified, the server can create the new device entry for the user and remove the pending entry from Firestore.

What’s Next?

In this post, we’ve explored what it takes to get started building a cloud service for smart home devices using the Google Cloud Platform and Firebase, and how these services make developing front-end applications easy while keeping everything secure end-to-end.

In the next post of this series, you will see how the groundwork we have laid enables us to quickly add the account linking and intent fulfillment necessary to connect your new device cloud with Smart Home Actions and the Google Assistant.

Building a Smart Home Cloud Service with Google was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

You already lost me, what’s a microcontroller?

Microcontrollers (MCUs) are simple, programmable, and fully integrated systems typically used for embedded control. In addition to a processor (CPU), they generally include all of the memory and peripheral interfaces necessary on a single chip. This simplicity and integration means that MCUs are relatively inexpensive and generally consume very little power. Many popular hardware development platforms, such as Arduino, are built on top of MCUs.

MCUs generally do not have the resources (such as a Memory Management Unit) to run a higher-level operating system like Linux or Android. Nor can they interface with high-speed peripherals like high-resolution cameras and displays. However, because the application code runs much closer “to the metal”, MCUs are very effective in real-time applications where timing is critical. Production MCUs often run some flavor of a real-time operating system (RTOS) to ensure tasks run in the exact amount of time required to ensure precise measurement and control.

All of these characteristics start to define applications where MCUs are a perfect fit…and where they aren’t.

The race to the cloud

Systems focused primarily (or entirely) on MCU hardware are based on what I’ll call the cloud first architecture. In this architecture, every edge device is connected directly to the internet (usually through WiFi), then provisioned and managed through a cloud service such as Google’s Cloud IoT Core. Inexpensive MCU platforms with built-in WiFi stacks, like the popular ESP8266, make designing systems like this very attractive.

In these systems, complex data analysis and decision making tasks are handled in the cloud back-end, while the device nodes perform data collection tasks or respond to remote control commands.

Overall, this is a nice balance. Hardware is inexpensive to replace and can run on small batteries for multiple years, and heavy compute resources are provided by cloud services that are easy to scale up to meet demand as the number of edge devices increases.

MCU hardware and a cloud-first architecture perform well in applications where bandwidth and network latency are less of a concern. Data payloads are small and can be uploaded to the cloud in batches. Here are some examples:

• Distributed sensor-based data collection
• Mobile asset monitoring and tracking

Living on the edge

There is a trend in IoT systems moving towards edge computing, enabled by the smartphone economy driving down the cost (and power consumption) of more capable hardware. There are four main reasons why applications perform computing tasks at the edge:

1. Privacy: Avoid sending all raw data to be stored and processed on cloud servers.
2. Bandwidth: Reduce costs associated with transmitting all raw data to cloud services.
3. Latency: Reaction time is critical and cannot be dependent on a cloud connection.
4. Reliability: The ability to operate even when the cloud connection is interrupted.

A great example of this in practice is the Google Home device. While the Google Assistant functionality is cloud-driven, the hotword detection happens locally on the device. This protects user privacy by avoiding uploads of audio data until the device knows you are talking to it, but it also eliminates the bandwidth that would have been consumed uploading raw audio to the cloud. Without edge computing, a device like this would not be feasible for consumer use.

This is also true in industrial automation systems where latency and reliability are critical. In these systems, one or more intermediate gateway devices act as an interface between local edge devices (which may be MCU-powered) and any cloud services.

Devices running Android Things are well-suited to gateway applications because they have the computational horsepower to locally apply data transformations and automation rules, paired with the SDK support to easily integrate with the Google Cloud Platform APIs.

There is also an upside here for MCU-powered nodes. In areas where the lowest power consumption is still critical, WiFi and Ethernet can often strain (or break) the power budget. If we remove the need to connect each device to a cloud service, we can substitute a more power-efficient local network transport, such as Bluetooth Low Energy or an 802.15.4 radio attached to a Thread mesh network.

Making IoT smarter

The advancements in both artificial intelligence (AI) and machine learning (ML) promise to have big effects on IoT systems in the coming years. The ability for ML algorithms to find patterns and make predictions based on data collected by devices will quickly become a necessary component to the success of IoT as the number of devices (and therefore the volume of data) continues to grow.

Systems built around a cloud-first architecture can already take advantage of services in Google’s Cloud AI suite, such as Cloud Vision and Cloud Speech, to enable machine learning. MCUs can interface with these services through REST APIs or indirectly via cloud functions and the bridges provided by Cloud IoT Core. The “heavy lifting”, however, must still be done in the cloud.

To truly scale these types of capabilities in IoT systems, we need to decentralize and push as much AI/ML as possible to the edge. This is where a platform like Android Things starts to really shine. In addition to having the necessary compute power, client libraries like the Google Assistant SDK and TensorFlow Lite enable edge devices to perform these complex tasks with very little developer integration work.

What’s next?

As I mentioned earlier, one of the advantages of MCUs is the ability to essentially control what happens on every clock cycle, providing real-time control over the I/O on the chip. This enables application firmware to implement custom protocols not directly supported by the hardware controllers on the processor.

We are starting to see chip vendors build these capabilities into their hardware, including both application processor (Cortex-A) and MCU (Cortex-M) cores within the same package. Architectures like this will enable developers to have a dedicated real-time core within the larger system, essentially providing the best of both worlds.

The right tools for the job

We’ve seen that demand for low latency, offline access, and enhanced machine learning capabilities is fueling a move towards decentralization with more powerful computing devices at the edge. Nevertheless, many distributed applications benefit more from a centralized architecture and the lowest cost hardware powered by MCUs. It all comes down to evaluating what your system’s needs truly are, and selecting the right tools for the job.

Computing at the Edge of IoT was originally published in Google Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

But there is one case that this local cache makes sense—a paused activity. If an activity is paused and then resumed, the default behavior will fetch again unless you serve a local copy like this one.