This page describes how to create and customize a push queue, and how to examine the contents of a queue.
Before you begin
Before you can use a task queue, note that you must first set up your queue configuration.
Using queue.yaml to create queues
To process a task, you must add it to a push queue. App Engine provides a
default push queue, named default, which is configured and
ready to use with default settings. If you want, you can just add all your tasks
to the default queue, without having to create and configure other queues.
If you need additional queues, or want to change the default queue
configuration, you can do so in the queue.yaml file for your application,
which you upload to App Engine. Note that you cannot create queues dynamically
in your program. You can create up to 10 queues for free applications, and up to
100 queues for billing-enabled applications.
This queue.yaml file defines two queues:
queue:
- name: queue-blue
target: v2.task-module
- name: queue-red
rate: 1/s
All tasks added to queue-blue are sent to the target module v2.task-module.
The refresh rate of queue-red is changed from the default 5/s to 1/s. Tasks
will be dequeued and sent to their targets at the rate of 1 task per second.
There are many other parameters that can be added to the queue.yaml file to
customize the behavior of a push queue. For more information, see the
queue.yaml
Defining the push queue processing rate
You can control the rate at which tasks are processed in each of your queues by
defining other directives, such as
rate,
bucket_size,
and
max_concurrent_requests.
The task queue uses token buckets to
control the rate of task execution. Each named queue has a token bucket that
holds tokens, up to the maximum specified by the bucket_size, or a maximum of
5 tokens if you don't specify the bucket size.
Each time your application executes a task, a token is removed from the bucket.
Your app continues processing tasks in the queue until the queue's bucket runs
out of tokens. App Engine refills the bucket with new tokens continuously based
on the rate that you specified for the queue.
If your queue contains tasks to process, and the queue's bucket contains tokens, App Engine simultaneously processes as many tasks as there are tokens. This can lead to bursts of processing, consuming system resources and competing with user-serving requests.
If you want to prevent too many tasks from running at once or to prevent
datastore contention, you use max_concurrent_requests.
The following samples shows how to set max_concurrent_requests to limit
tasks and also shows how to adjust the bucket size and rate based on your
application's needs and available resources:
queue:
- name: optimize-queue
rate: 20/s
bucket_size: 40
max_concurrent_requests: 10
Setting storage limits for all queues
You can use queue.yamltotal_storage_limit
at the top level:
# Set the total storage limit for all queues to 120MB
total_storage_limit: 120M
queue:
- name: foo
rate: 35/s
The value is a number followed by a unit: B for bytes, K for kilobytes, M
for megabytes, G for gigabytes, T for terabytes. For example, 100K
specifies a limit of 100 kilobytes. If adding a task would cause the queue to
exceed its storage limit, the call to add the task will fail. The default limit
is 500M (500 megabytes) for free apps. For billed apps there is no limit until
you explicitly set one. You can use this limit to protect your app from a fork
bomb
programming error in which each task adds multiple other tasks during its
execution. If your app is receiving errors for insufficient quota when adding
tasks, increasing the total storage limit can help. If you are using this
feature, we strongly recommend setting a limit that corresponds to the storage
required for several days' worth of tasks. In this way, your app is robust to
its queues being temporarily backed up and can continue to accept new tasks
while working through the backlog while still being protected from a fork bomb
programming error.
Monitoring queues in the Cloud Platform Console
The Task Queues page in the Cloud Platform Console. displays information about all the task queues in your application.
-
Visit the Task Queues page in the Cloud Platform Console and select the Push Queues tab in the menu bar at the top of the page.
-
The Push Queues tab lists all of the queues in the application. Clicking on a queue name brings up the Task Queue Details page where you can see all of the tasks in the selected queue.
What's next
- Learn about creating tasks
