Public
ISCA paper preprint about Google's Tensor Processing Unit
Paper: https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view
Blog post by +Norm Jouppi: https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html
Last June at Google I/O, +Sundar Pichai showed an example of a new type of custom ASIC that Google had developed to accelerate machine learning workloads, called a Tensor Processing Unit (TPU), but didn't give very many details. The TPU is used to run large neural networks very efficiently and with low latency throughout many Google products, including Search, Photos, Translate, and also powered the AlphaGo system used during the match against Lee Sedol in Korea last March, and offers 92 trillion operations per second (TOPs) per chip with a modest power budget. I'm happy to announce that we now have a detailed paper In-Datacenter Performance Analysis of a Tensor Processing Unit that will appear in this year's International Symposium on Computer Architecture (ISCA) conference in Toronto in June. Today we've published a pre-print of the paper and a companion blog post, and +David Patterson will be giving a talk about the TPU at the Computer History Museum in Mountain View this afternoon (https://sites.google.com/corp/view/naeregionalsymposium: sadly no more space is available).
Various news articles:
https://www.nextplatform.com/2017/04/05/first-depth-look-googles-tpu-architecture/
https://www.wired.com/2017/04/building-ai-chip-saved-google-building-dozen-new-data-centers/
Hacker News discussion: https://news.ycombinator.com/item?id=14043059
Paper: https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view
Blog post by +Norm Jouppi: https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html
Last June at Google I/O, +Sundar Pichai showed an example of a new type of custom ASIC that Google had developed to accelerate machine learning workloads, called a Tensor Processing Unit (TPU), but didn't give very many details. The TPU is used to run large neural networks very efficiently and with low latency throughout many Google products, including Search, Photos, Translate, and also powered the AlphaGo system used during the match against Lee Sedol in Korea last March, and offers 92 trillion operations per second (TOPs) per chip with a modest power budget. I'm happy to announce that we now have a detailed paper In-Datacenter Performance Analysis of a Tensor Processing Unit that will appear in this year's International Symposium on Computer Architecture (ISCA) conference in Toronto in June. Today we've published a pre-print of the paper and a companion blog post, and +David Patterson will be giving a talk about the TPU at the Computer History Museum in Mountain View this afternoon (https://sites.google.com/corp/view/naeregionalsymposium: sadly no more space is available).
Various news articles:
https://www.nextplatform.com/2017/04/05/first-depth-look-googles-tpu-architecture/
https://www.wired.com/2017/04/building-ai-chip-saved-google-building-dozen-new-data-centers/
Hacker News discussion: https://news.ycombinator.com/item?id=14043059
View 12 previous comments
Yes. TPU, K80, and Maxwell are in 28 nm semiconductor technology, but the TPU and K80 are datacenter class as they protect on-chip SRAM, but Maxwell doesn't, so doesn't make sense to compare since we can't put them in the datacenter. For AVX2, (with a lot of work) we did get a datapoint that we could quote. Can't do that for Maxwells, since they're not in our datacenters for the reason stated.
That being said, P40 is much faster than Maxwell.15w
Thanks for clarifying! I was not sure/forgot, but it makes sense that Maxwell ended up with protection on the DRAM/bus "strapped on" (given that it seemed like a very late after-thought).
That AVX2 work required was significant hints, I assume, that there is no CPU SIMD backend at all for Tensorflow (not even internally)?15w
+Jeff Dean, is there a commercial plan to sell this TPU or is actually only for internal use?15w
Given that this paper is released in 2017 and compares against tech that is 2 generations old and at the time of release there's HW like P40 available with 50 INT8 TOPS for inference I frankly find this very late paper announcement very deliberately misleading. So in 2017 you release this "new" publication where you say "we were better in 2015". Who cares? There must be a special codename for this particularly twisted logical fallacy. You know very well that you get headlines like "TPU is 30x faster than GPU" based on these completely irrelevant facts. Therefore you are misleading/cheating your customers.9w- +Andrei Pokrovsky While generally you have a fair point this work is on the crossroads of research, marketing, product development in a highly competitive field, so we need to take extra caution to judge things based on their merit.
While the might can grab headlines, arguably that's not the main role of a research paper. Judging the research itself, the criticism you raise is misguided, I have to say. The results of the research has been accepted for a conference and publication in the proceedings of this and if you wish to criticize it, I think it is more productive to stick to talking about the merits of the work rather than tangential personal opinions.
Related to the missing comparisons with Pascal, please take the time to read the paper and related comments here: the cards were not out at the time of publication. While the timing an omission is unfortunate, I doubt it's very relevant, the arch presented here is previous-gen (comparing to previous-gen data-center GPUs) and the TPU2 has already been released anyway.9w 
+Andrei Pokrovsky As the paper discusses briefly, "The new 16-nm, 1.5GHz, 250W P40 datacenter GPU can perform 47 Tera 8-bit ops/sec, but was unavailable in early 2015, so isn’t contemporary with our three platforms." .
Although the paper was submitted to the ISCA conference in late 2016 and put on Arxiv in 2017, the work it describes is from a system that has been deployed since early 2015, and the typical way that one does performance comparisons in computer architecture is against the other architectural choices available at the same time as the work is being done. If you don't do that, it's hard to tease apart the effects of, say, fabrication process improvements (e.g. 16nm vs. 28nm) that weren't available at the time, vs. architectural design choices.
We also aren't trying to appeal to customers with this paper: this is a research publication describing what we found to be a useful design that is different than CPUs and GPUs, and to make comparisons of the approach that help computer architects in the future. Indeed, with this first generation TPU, our only customers are Google and Alphabet's internal users, as we don't have any products that involve direct access to a TPU. Customers are indirctly using TPUs through much higher level interfaces (like Google Search, or Google Translate, or Google Photos), but that's fairly indirect.9w
