Configuring Datastore Indexes

The App Engine datastore uses indexes for every query your application makes. These indexes are updated whenever an entity changes, so the results can be returned quickly when the app makes a query. To do this, the datastore needs to know in advance which queries the application will make. You specify which indexes your app needs in a configuration file. The development server can generate the datastore index configuration automatically as you test your app.

This guides provides instructions for creating your indexes and managing them. For more background information, see Datastore Indexes.

Creating Datastore indexes

Every datastore query made by an application needs a corresponding index. Indexes for simple queries, such as queries over a single property, are created automatically. Indexes for complex queries must be defined in a configuration file named index.yaml. This file is uploaded with the application to create indexes in the datastore.

The following is an example of an index.yaml file:

indexes:

- kind: Cat
  ancestor: no
  properties:
  - name: name
  - name: age
    direction: desc

- kind: Cat
  properties:
  - name: name
    direction: asc
  - name: whiskers
    direction: desc

- kind: Store
  ancestor: yes
  properties:
  - name: business
    direction: asc
  - name: owner
    direction: asc

Creating indexes using the development server

The development web server (dev_appserver.py) automatically adds items to this file when the application tries to execute a query that needs an index that does not have an appropriate entry in the configuration file.

In the development server, if you exercise every query that your app will make, the development server will generate a complete list of entries in the index.yaml file.

When the development web server adds a generated index definition to index.yaml, it does so below the following line, inserting it if necessary:

# AUTOGENERATED

The development web server considers all index definitions below this line to be automatic, and it might update existing definitions below this line as the application makes queries.

Creating indexes manually

You can add indexes manually to the index.yaml file or you can edit existing entries. For manually controlled queries, you must add them above the line that is commented with AUTOGENERATED.

All index definitions above this line are considered to be under manual control, and are not updated by the development web server. The development server will only make changes below the line, and will only do so if the complete index.yaml file does not describe an index that accounts for a query executed by the application. To take control of an automatic index definition, move it above this line.

See the index.yaml reference for syntax information.

Updating indexes

When you upload an application using appcfg.py update, the update includes the app's index.yaml file. If the index.yaml file defines an index that doesn't exist yet on App Engine, App Engine creates the new index. Depending on how much data is already in the datastore that needs to be mentioned in the new index, the process of creating the index may take a while. If the app performs a query that requires an index that hasn't finished building yet, the query will raise an exception.

To prevent this, you must ensure that the new version of the app that requires a new index is not the live version of the application until the indexes finish building. One way to do this is to give the app a new version number in app.yaml whenever you add or change an index in index.yaml. The app is uploaded as a new version, and does not become the default version automatically. When your indexes have finished building, you can change the default version in the Cloud Platform Console.

Another way to ensure that new indexes are built before the new app goes live is to upload the index.yaml configuration separately before uploading the app. To upload only the index configuration for an app, use the following command:

appcfg.py update_indexes myapp/

In the Cloud Platform Console, you can check the status of the app's indexes.

Deleting unused indexes

When you change or remove an index from index.yaml, the original index is not deleted from App Engine automatically. This gives you the opportunity to leave an older version of the app running while new indexes are being built, or to revert to the older version immediately if a problem is discovered with a newer version.

When you are sure that old indexes are no longer needed, you can delete them from App Engine using the following command:

appcfg.py vacuum_indexes myapp/

This command deletes all indexes for the app that are not mentioned in the local version of index.yaml.