Official Blog
Insights from Googlers into our products, technology, and the Google culture
Using large-scale brain simulations for machine learning and A.I.
June 26, 2012
You probably use
machine learning
technology dozens of times a day without knowing it—it’s a way of training computers on real-world data, and it enables high-quality
speech recognition
, practical
computer vision
, email
spam blocking
and even
self-driving cars
. But it’s far from perfect—you’ve probably chuckled at poorly transcribed text, a bad translation or a misidentified image. We believe machine learning could be far more accurate, and that smarter computers could make everyday tasks much easier. So our research team has been working on some new approaches to large-scale machine learning.
Today’s machine learning technology takes significant work to adapt to new uses. For example, say we’re trying to build a system that can distinguish between pictures of cars and motorcycles. In the standard machine learning approach, we first have to collect tens of thousands of pictures that have already been labeled as “car” or “motorcycle”—what we call
labeled data
—to train the system. But labeling takes a lot of work, and there’s comparatively little labeled data out there.
Fortunately, recent research on
self-taught learning
(PDF) and
deep learning
suggests we might be able to rely instead on
unlabeled data
—such as random images fetched off the web or out of YouTube videos. These algorithms work by building artificial neural networks, which loosely simulate neuronal (i.e., the brain’s) learning processes.
Neural networks are very computationally costly, so to date, most networks used in machine learning have used only 1 to 10 million connections. But we suspected that by training much larger networks, we might achieve significantly better accuracy. So we developed a distributed computing infrastructure for training large-scale neural networks. Then, we took an artificial neural network and spread the computation across 16,000 of our
CPU cores
(in
our data centers
), and trained models with more than 1 billion connections.
We then ran experiments that asked, informally: If we think of our neural network as simulating a very small-scale “newborn brain,” and show it YouTube video for a week, what will it learn? Our hypothesis was that it would learn to recognize common objects in those videos. Indeed, to our amusement, one of our artificial neurons learned to respond strongly to pictures of... cats. Remember that this network had never been told what a cat was, nor was it given even a single image labeled as a cat. Instead, it “discovered” what a cat looked like by itself from only unlabeled YouTube stills. That’s what we mean by self-taught learning.
One of the neurons in the artificial neural network, trained from still frames from unlabeled YouTube videos, learned to detect cats.
Using this large-scale neural network, we also significantly improved the state of the art on a standard image classification test—in fact, we saw a 70 percent relative improvement in accuracy. We achieved that by taking advantage of the vast amounts of unlabeled data available on the web, and using it to augment a much more limited set of labeled data. This is something we’re really focused on—how to develop machine learning systems that scale well, so that we can take advantage of vast sets of unlabeled training data.
We’re reporting on these experiments, led by Quoc Le, at
ICML
this week. You can get more details in
our Google+ post
or read
the full paper
(PDF).
We’re actively working on scaling our systems to train even larger models. To give you a sense of what we mean by “larger”—while there’s no accepted way to compare artificial neural networks to biological brains, as a very rough comparison an adult human brain has around 100
trillion
connections. So we still have lots of room to grow.
And this isn’t just about images—we’re actively working with other groups within Google on applying this artificial neural network approach to other areas such as speech recognition and natural language modeling. Someday this could make the tools you use every day work better, faster and smarter.
Posted by Jeff Dean, Google Fellow and Andrew Ng, Visiting Faculty
Labels
accessibility
39
acquisition
26
ads
131
Africa
19
Android
59
apps
417
April 1
4
Asia
39
books + book search
48
commerce
12
computing history
7
crisis response
32
culture
11
developers
120
diversity
33
doodles
65
education and research
144
entrepreneurs at Google
14
Europe
46
faster web
16
free expression
61
google.org
69
googleplus
50
googlers and culture
203
green
101
Latin America
18
maps and earth
192
mobile
125
online safety
19
open source
19
photos
38
policy and issues
139
politics
69
privacy
66
recruiting and hiring
32
scholarships
31
search
502
search quality
24
search trends
118
security
36
small business
31
user experience and usability
41
youtube and video
140
Archive
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2007
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2006
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2005
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2004
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Feed
Follow @google
Follow
Give us feedback in our
Product Forums
.
Follow
