[2/2] How did we get to where we are in AI? (AI column)

Continue. story Jeff Dean's performance should be told, and which key whitepapers came out after Attention Is All You Need We will continue to share our findings on where we are today.

2017 - Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer An article about the now common approach with discharged networks or a mixture of experts (MOE). The article shows how conditional computation can be used to build “outrageously large” networks with tens or hundreds of billions of parameters, almost without increasing computational costs compared to conventional models.

2018 - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding The authors showed that one large bidirectional transformer model can be pre-trained once on raw text, and then with minimal modifications to be trained for dozens of different NLP tasks, obtaining state-of-the-art without special architectures for each task.

2021 - An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale In this paper, the authors applied the transformer approach to image classification. The interesting thing is that you can do this without CNN -- just take a pure transformer and feed it an image as a sequence of fixed-size elements. (the same16×16 title).

2022 - Pathways: Asynchronous Distributed Dataflow for ML The article is about asynchronous distributed data flow, when the calculation is set as a graph of operators exchanging futures. And then a single controller can simultaneously plan and run heterogeneous tasks on the TPU cluster, hiding dependencies in the data-plane and simplifying the software model and resource management. In general, this provides scaling of computation at Google scale.

2022 - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models The discovery was that models were better at solving problems when asked to “think step by step.” If you show the model an example of reasoning, it begins to generate intermediate steps of calculations, which dramatically improves accuracy in mathematics and logic.

Finally, Jeff mentions the release of the new model Gemini 3.0, which combines all previous achievements that helped her knock out SOTA on many benches.

As a result, if you watch the lecture and flip through the whitepaper, you can draw approximately the following conclusions: - Scale matters.. Recent progress 15 For years it was provided not only with new ideas, but also with rough computing power. The increase in the number of parameters and volume of data invariably led to the emergence of new abilities. (emergent capabilities)Which little models didn't have. (For example, understanding humor or solving problems in physics). Iron specialization is inevitable.. Universal processors (CPU) They are no longer the driver of progress. The future is for specialized chips (TPU)sharpened for low-precision linear algebra. Energy efficiency is becoming a key constraint for further model growth. - Diluted models (Sparse Models) - the path to efficiency. Dean highlights transition to architecture n (similar)where only a small part of the neural network is activated to process one request (1-5%). This makes it possible to make models colossal in terms of “knowledge”, but fast in operation. - Multimodality as a standard. AI is no longer just “textual.” Modern systems natively understand and generate video, audio and images. An example from the video: a model can read handwritten recipes in different languages, translate them, generate pictures of dishes and write code for a finished website. AI as an assistant in science and creativity. The main positive impact of AI is expected in accelerating scientific discoveries (AlphaFold, materials science) Lowering the entry threshold for complex skills (programming, design). There are risks in the development of this technology. For example, generated content becomes indistinguishable from real content, and technical and social protection mechanisms are needed.

#AI #ML #Software #Engineering #Architecture #Infrastructure #Data