[1/2] How did we get to where we are in AI? (AI column)
A couple of weeks ago, Jeff Dean. (Jeff Dean), Chief Data Science at Google, performer Stanford AI Club, where he gave a retrospective 15+ years of development of artificial intelligence. He explained as a combination of three factors: scaling computation. (scale)New algorithms and specialized hardware have led to modern large multimodal models such as Gemini. 3.0. Jeff is a legendary figure in the world of CS and software engineering. 1999 He is now Chief Data Science, and has a hand in MapReduce, BigTable, Spanner, TensorFlow and now Gemini. 3. Below is a list of whitepapers that Jeff Dean has identified as key points of technology development. (In some cases, he was a co-author.)
2012 - Large Scale Distributed Deep Networks Until then, neural networks were trained on local machines. Dean and team created a software architecture for distributed learning across thousands of CPUs. This allowed to train the models in 50-100 times more than anyone before. The whitepaper itself talks about the use of data parallelism and models. (Data/Model Parallelism) and asynchronous stochastic gradient descent.
2012 - Building high-level features using large scale unsupervised learning In this experiment, the neural network "watched" 10 Millions of random YouTube footage without markup. As a result, the model independently learned to recognize the concepts. (For example, cats, human faces) Just watching the data. This has proven the effectiveness of unsupervised learning on a large scale.
2013 - Distributed Representations of Words and Phrases and their Compositionality In this whitepaper, we talked about the Word2Vec algorithm for building vector representations of words and transitioning the perception of a word as a discrete value to a vector in a multidimensional space. In the end, it turned out that words with similar meanings are next to each other in vector space. Moreover, arithmetic operations on vectors preserve semantics. (King - Man + Woman = Queen).
2014 - Sequence to Sequence Learning with Neural Networks The authors presented the Seq2Seq algorithm using recurrent networks (LSTM) for sequence translation tasks. It worked something like this: one network encodes the input phrase. (for example in English) the vector, and the other decodes it at the output (for example in French). This approach worked well and became the basis for machine translation for years to come.
2015 - Distilling the Knowledge in a Neural Network The authors described a method of compressing knowledge of a huge model into a small and fast one. The concept was that a small model ("student") It learns not only from the correct answers, but also by imitating the probability distribution of a large model. ("teacher"). This allows you to run powerful AI on mobile devices.
2017 - In-Datacenter Performance Analysis of a Tensor Processing Unit The story about how the guys at Google realized that you need to come up with something instead of a CPU and a GPU to work neural networks. So the guys decided to do TPU. (tensor process unit)Whose story am I? handler separately. Made in 2015 And they started it, and they told me about it. 2017. And then Jeff remembers in a lecture about a configurable supercomputer. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingswhat they did in 2017And it was told in a whitepaper in 2023
2017 - Attention Is All You Need Famous article that brought to the world the revolutionary architecture of transformers, which allowed to abandon RNN (recurrent) attentionalism (Self-Attention). The concept is that now the model can "look" at all the words in the sentence at the same time, and not in turn - this allows you to balance where it pay attention + there is parallelism at the input. This provided a multiple increase in learning speed and quality, becoming the basis for all modern LLMs. (GPT, Gemini, Claude).
Continuation of the review in second.
#AI #ML #Software #Engineering #Architecture #Infrastructure #Data