This document describes how to monitor BigQuery resources using Stackdriver Monitoring, including creating charts and alerts for BigQuery metrics.
Before you begin
-
Before you use Stackdriver Monitoring, ensure that you have the following:
- A Google Cloud Platform account.
- A BigQuery project with billing enabled.
One way to ensure that you have both is to complete the Quickstart Using the Web UI.
-
If you do not have a Stackdriver account, follow the steps in the Create a Stackdriver Account section of the Stackdriver Setting Up Accounts documentation.
Viewing and creating dashboards, charts, and alerts
Viewing the Stackdriver dashboard
To use Stackdriver to monitor your BigQuery project:
-
In the Google Cloud Platform Console, select Stackdriver > Monitoring, or use the following button:
-
Select the name of your project if it is not already selected at the top of the page.
-
To view BigQuery resources, select Resources > BigQuery:
-
Selecting BigQuery takes you to the default BigQuery page. On this page you’ll see a list of tables, Events and Incident reporting that are user configurable as well as charts of Projects Metrics or Dataset Metrics.
Visualizing Slots Available and Slots Allocated
To visualize the slots available and slots allocated to your project:
-
Go to the Stackdriver dashboard for BigQuery described in Viewing the Stackdriver dashboard.
-
On the Stackdriver dashboard for BigQuery, scroll down to the chart named Slot Utilization.
The Slot Utilization chart appears on both the main Stackdriver default dashboard and the Stackdriver dashboard for BigQuery.
Creating a dashboard and chart
Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:
-
In the top menu of the Stackdriver Monitoring Console, select Dashboards > Create....
-
Click Add Chart. You see the Add Chart page:
-
In the Resource Type menu, select "BigQuery".
-
In the Metric Type menu, select "Query Time - 50th Percentile". You see the chart data in the Preview section.
-
Click Save.
Creating an alert
Create an alert by defining a condition and configuring the notification settings.
-
In the top menu of the Stackdriver Monitoring Console, select Alerting > Create a Policy, which opens the Create new Alerting Policy page.
-
In the Conditions section, click Add Condition.
-
The Select Condition Type page offers several types of conditions. For this example, choose Metric Threshold by clicking on its associated Select button.
-
On the Add Metric Threshold Condition page:
- Set RESOURCE TYPE to "BigQuery".
- Set APPLIES TO to "All".
- Set IF METRIC to "Query Time (50th Percentile)".
- Set THRESHOLD to a number, such as 8.
- Leave all other default values and click Save Condition.
-
On the Create new Alerting Policy page, click Add Notification to configure your notification settings. You can be notified by email, SMS, and several other options.
-
You can customize the notification message by clicking the Add Documentation button in the Documentation section.
-
Finally, name your policy and click Save Policy.
Metrics available for visualization
The following metrics are available, time delayed up to several hours.
| Resource Type | Name | Units | Description |
|---|---|---|---|
| BigQuery | Scanned Bytes | Bytes per Minute | Number of bytes scanned |
| BigQuery | Scanned Bytes Billed | Bytes per Minute | Number of bytes sent for billing
The Scanned Bytes and Scanned Bytes Billed can differ for a couple reasons.
|
| BigQuery | Query Time |
seconds | Non-cached query execution times (s) |
| BigQuery | Slots Allocated | slots | Number of slots allocated to the project at any given moment. This can
also be thought of as the number of slots being utilized by that project.
Slots are allocated per billing account and multiple projects can share the same reservation of slots. |
| BigQuery | Slots Available | slots | Total number of slots available to the project.
If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted. |
| BigQuery Dataset | Stored Bytes | Bytes | Bytes stored in the dataset - this is broken down by table for the 100 largest tables in the dataset. |
| BigQuery Dataset | Tables | Tables | Number of Tables in the Dataset |
| BigQuery Dataset | Uploaded Billed Bytes | Bytes per minute | Number of bytes uploaded to any table in the dataset that were billed |
| BigQuery Dataset | Uploaded Bytes | Bytes per minute | Number of bytes uploaded to any table in the dataset
The Uploaded Billed Bytes and Uploaded Bytes Billed can differ for a couple reasons.
|
| BigQuery Dataset | Uploaded Rows | Rows per minute | Number of records uploaded to any table in the dataset |
Known issues
-
If no queries are running then no data will be returned for Slots Allocated or for Slots Available or for any query related variables. Zoom out to see data.
-
If queries are running in both the US and the EU then Slots Allocated and Slots Available might be incorrect.
-
Slots Allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). As such, zooming in and out can change the value of Slots Allocated. Zooming in to a time window of 1 hr of less shows the true values of Slots Allocated. At this range for any time visible on the chart, avg(Slots Allocated) = Slots Allocated.
-
Very few BigQuery customers use sub-reservations. If your project uses sub-reservations and you are using Stackdriver to monitor your utilization, please contact Cloud Support for details on how Stackdriver displays Slots Allocated and Slots Available.
-
The data in Stackdriver charts pertains only to the selected project. Multi-project monitoring has not yet been implemented.
Sending feedback
There is a Send Feedback button at the bottom right of all Stackdriver windows. Please send us suggestions and bug reports.
