Downloads

In addition to the HTTP Archive source code being open source, all of the data gathered is available as downloadable MySQL dumps. These dumps are also available in Google BigQuery thanks to Ilya Grigorik. For more information see Ilya's blog post HTTP Archive + BigQuery = Web Performance Answers and video. And checkout all the custom queries shared on BigQueri.es.

MySQL Dumps

The results from each crawl are saved as MySQL dump files in both MySQL format and CSV format. Dumps are made for both the desktop and mobile crawls. The dump files do not contain the commands to create the MySQL database and tables. To restore a dump:

  1. Import the schema to create the tables.
    cat httparchive_schema.sql | mysql -u MYSQL_USERNAME -p -h MYSQL_HOSTNAME MYSQL_DB
  2. Import the desired dump file. Use this command for MySQL-formatted dumps:
    gunzip -c MYSQLDUMP_FILE.gz | mysql -u MYSQL_USERNAME -p -h MYSQL_HOSTNAME MYSQL_DB
    For CSV-formatted dumps, gunzip the file and rename it to either pages.csv or requests.csv, then run this command:
    mysqlimport --local --fields-optionally-enclosed-by="\"" --fields-terminated-by=, --user=MYSQL_USERNAME -p MYSQL_DB MYSQLDUMP_FILE.csv
  3. If you want to run a private instance of the source code, you need to also import the stats and crawls dumps.

Files

These files define the schema and the meta-level tables:

There's a download file for each type of crawl: