Attention Is All You Need

"Attention Is All You Need" is a research paper by researchers at Google (Google Brain and Google Research), posted to arXiv in June 2017 (arXiv:1706.03762). It proposes the Transformer, a sequence transduction architecture that dispenses with recurrence and convolutions and relies entirely on attention mechanisms in an encoder-decoder configuration.

The paper reports 28.4 BLEU on WMT 2014 English-to-German translation, improving over prior best results by more than 2 BLEU, and a single-model 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs. It also shows the architecture generalizing to English constituency parsing. The model is more parallelizable and cheaper to train than the recurrent and convolutional models it was compared against.

The Transformer became the basis of most modern large language models and many other model families. The paper is freely available on arXiv. The reference implementation was released in Google's Tensor2Tensor library (Apache-2.0), which is now deprecated in favor of Trax. The paper is not a product or model with weights to run.