Google Research Blog
The latest news from Research at Google
Google Cluster Data
Thursday, January 07, 2010
Posted by Joseph L. Hellerstein, Manager of Google Performance Analytics
Google faces a large number of technical challenges in the evolution of its applications and infrastructure. In particular, as we increase the size of our compute clusters and scale the work that they process, many issues arise in how to schedule the diversity of work that runs on Google systems.
We have distilled these challenges into the following research topics that we feel are interesting to the academic community and important to Google:
Workload characterizations:
How can we characterize Google workloads in a way that readily generates synthetic work that is representative of production workloads so that we can run stand alone benchmarks?
Predictive models of workload characteristics:
What is normal and what is abnormal workload? Are there "signals" that can indicate problems in a time-frame that is possible for automated and/or manual responses?
New algorithms for machine assignment:
How can we assign tasks to machines so that we make best use of machine resources, avoid excess resource contention on machines, and manage power efficiently?
Scalable management of cell work:
How should we design the future cell management system to efficiently visualize work in cells, to aid in problem determination, and to provide automation of management tasks?
To aid researchers in addressing these questions in a realistic manner, we will provide data from Google production systems. The initial focus of these data will be workload characterization. Details of the data can be found
here
. The data are structured as follows:
Time (int) - time in seconds since the start of data collection
JobID (int) - Unique identifier of the job to which this task belongs
TaskID (int) - Unique identifier of the executing task
Job Type (0, 1, 2, 3) - class of job (a categorization of work)
Normalized Task Cores (float) - normalized value of the average number of cores used by the task
Normalized Task Memory (float) - normalized value of the average memory consumed by the task
We solicit your
feedback
in terms of: (a) the quality and content of the data we are providing; (b) technical approaches and/or results related to the topics above; and (c) other research topics that you feel Google should be addressing in the area of Cloud Computing (along with details of the data required to address these topics).
Labels
accessibility
ACL
ACM
Acoustic Modeling
Adaptive Data Analysis
ads
adsense
adwords
Africa
Android
API
App Engine
App Inventor
April Fools
Audio
Australia
Automatic Speech Recognition
Awards
Cantonese
China
Chrome
Cloud Computing
Collaboration
Computational Photography
Computer Science
Computer Vision
conference
conferences
Conservation
correlate
Course Builder
crowd-sourcing
CVPR
Data Center
data science
datasets
Deep Learning
distributed systems
Diversity
Earth Engine
economics
Education
Electronic Commerce and Algorithms
EMEA
EMNLP
Encryption
entities
Entity Salience
Environment
Exacycle
Faculty Institute
Faculty Summit
Flu Trends
Fusion Tables
gamification
Genomics
Gmail
Google Books
Google Drive
Google Science Fair
Google Sheets
Google Translate
Google Voice Search
Google+
Government
grants
HCI
Health
High Dynamic Range Imaging
ICML
ICSE
Image Annotation
Image Classification
Image Processing
Inbox
Information Retrieval
internationalization
Internet of Things
Interspeech
IPython
Journalism
jsm
jsm2011
K-12
KDD
Klingon
Korean
Labs
Linear Optimization
localization
Machine Hearing
Machine Intelligence
Machine Learning
Machine Translation
MapReduce
market algorithms
Market Research
ML
MOOC
NAACL
Natural Language Processing
Natural Language Understanding
Network Management
Networks
Neural Networks
Ngram
NIPS
NLP
open source
operating systems
Optical Character Recognition
osdi
osdi10
patents
ph.d. fellowship
PiLab
Policy
Professional Development
Public Data Explorer
publication
Publications
Quantum Computing
renewable energy
Research
Research Awards
resource optimization
Search
search ads
Security and Privacy
SIGCOMM
SIGMOD
Site Reliability Engineering
Software
Speech
Speech Recognition
statistics
Structured Data
Systems
TensorFlow
Translate
trends
TTS
TV
UI
University Relations
UNIX
User Experience
video
Vision Research
Visiting Faculty
Visualization
VLDB
Voice Search
Wiki
wikipedia
WWW
YouTube
Archive
Archive
December 2015 ( 2 )
November 2015 ( 2 )
October 2015 ( 2 )
September 2015 ( 4 )
August 2015 ( 12 )
July 2015 ( 9 )
June 2015 ( 6 )
May 2015 ( 3 )
April 2015 ( 3 )
March 2015 ( 4 )
February 2015 ( 4 )
January 2015 ( 1 )
December 2014 ( 8 )
November 2014 ( 3 )
October 2014 ( 7 )
September 2014 ( 8 )
August 2014 ( 4 )
July 2014 ( 4 )
June 2014 ( 2 )
May 2014 ( 1 )
April 2014 ( 4 )
March 2014 ( 4 )
February 2014 ( 5 )
January 2014 ( 2 )
December 2013 ( 3 )
November 2013 ( 9 )
October 2013 ( 2 )
September 2013 ( 5 )
August 2013 ( 2 )
July 2013 ( 6 )
June 2013 ( 7 )
May 2013 ( 5 )
April 2013 ( 3 )
March 2013 ( 4 )
February 2013 ( 4 )
January 2013 ( 1 )
December 2012 ( 4 )
October 2012 ( 4 )
September 2012 ( 3 )
August 2012 ( 9 )
July 2012 ( 9 )
June 2012 ( 7 )
May 2012 ( 7 )
April 2012 ( 2 )
March 2012 ( 7 )
February 2012 ( 3 )
January 2012 ( 4 )
December 2011 ( 5 )
November 2011 ( 2 )
September 2011 ( 3 )
August 2011 ( 4 )
July 2011 ( 9 )
June 2011 ( 6 )
May 2011 ( 4 )
April 2011 ( 4 )
March 2011 ( 5 )
February 2011 ( 5 )
January 2011 ( 4 )
December 2010 ( 7 )
November 2010 ( 2 )
October 2010 ( 9 )
September 2010 ( 7 )
August 2010 ( 2 )
July 2010 ( 7 )
June 2010 ( 3 )
May 2010 ( 2 )
April 2010 ( 1 )
March 2010 ( 1 )
February 2010 ( 1 )
January 2010 ( 2 )
December 2009 ( 8 )
November 2009 ( 4 )
August 2009 ( 4 )
July 2009 ( 5 )
June 2009 ( 5 )
May 2009 ( 4 )
April 2009 ( 6 )
March 2009 ( 3 )
February 2009 ( 1 )
January 2009 ( 4 )
December 2008 ( 1 )
November 2008 ( 1 )
October 2008 ( 1 )
September 2008 ( 1 )
July 2008 ( 1 )
May 2008 ( 3 )
April 2008 ( 1 )
March 2008 ( 1 )
February 2008 ( 1 )
October 2007 ( 1 )
September 2007 ( 2 )
August 2007 ( 1 )
July 2007 ( 1 )
June 2007 ( 2 )
February 2007 ( 2 )
December 2006 ( 1 )
November 2006 ( 1 )
September 2006 ( 1 )
August 2006 ( 1 )
July 2006 ( 1 )
June 2006 ( 2 )
April 2006 ( 3 )
March 2006 ( 4 )
February 2006 ( 1 )
Feed
Follow @googleresearch
Give us feedback in our
Product Forums
.